Midi Device Schema Specifications

This document proposes a solution for describing the midi System Exclusive implementation of any midi capable device.

 

Date: 1 October 2006

Author: Marc Jacobi (obiwanjacobi@hotmail.com)

Document status: draft, version: 0.3

 

Content:

Abstract. 1

Overview.. 2

Requirements. 3

Schema. 4

Declarations 5

The Type System. 9

Record Base Types 11

Implementation details. 11

Schema Object Model 12

Data Object Model 12

Data Converters 13

Data Transformation. 13

Resources. 14

 

Abstract

The goal of the Midi Device Schema is to unambiguously describe the midi capabilities of a midi enabled hardware (or virtual) device focusing on the System Exclusive implementation of that device. This description can then be interpreted by software and used to provide structured System Exclusive parsing and compiling of Midi messages. Such software would relieve an (midi) application developer from having to understand and implement all the details of the devices he or she wishes to support in their software.

 

Over the years the System Exclusive part of the midi protocol has been proven to be a pain. Manufacturers implement their System Exclusive messages in what is in their view the best possible way (best meaning cheap in most cases). Because the System Exclusive messages has no predefined structure overall (other than start and end markers), they need to be handled specifically and separately in the application code of midi software. This leads to application developers having to write specialized code for each device and not being able to support all devices. If, as a user, your midi device was not on the supported devices list, you could not use the software.

 

The Midi Device Schema aims to change that. In order to tackle this problem, the proposed solution uses metadata to describe the Midi System Exclusive implementation of a specific midi device in a schema.

 

Overview

Although this specification only deals with the textual Midi Device Schema declarations I like to give you a baseline-architecture to position the explanation of schema content in.

 

The following figure displays this architecture using a receive scenario.

The Data Object Model component is instantiated for one specific device for which to encipher the midi data (both send and receive). The Midi Device Schema (file or otherwise) is located and parsed into the Schema Object Model. This object model is a read-only, in-memory representation of the Midi Device Schema. The Midi Device Schema is expressed using the Midi Type System and optional custom types. The Data Object Model component will create the necessary Data Converters for this particular Midi Device Schema. Next the application code routes the incoming Midi Data Stream through the Data Object Model which in turn calls into the Data Converters component to convert the physical midi data into a logical in-memory representation that is passed to an Application Component for further processing in the application.

 

Figure 1: Basic architecture

 

Sending logical midi data from the application to the midi device would require the application to pass this data into the Data Object Model which in turn will call into the Data Converter component to transform the data into physical midi data according to the Midi Device Schema.

 

All behavior of transforming midi data from physical to logical and visa versa is driven by the declarations in the Midi Device Schema.

 

Requirements

The requirements laid out for the Midi Device Specification are as follows:

 

1.x

Global Requirements

2.x

Transformation Requirements

3.x

Extensibility Requirements

 

These requirements represent some of the variations encountered while doing a short analysis of several midi Device System Exclusive implementations (Roland and Yamaha).

 

#

Requirement

Solution

1.0

All oddities of packaging 8-bit data in 7-bit Midi System Exclusive must be abstracted from the developer.

Logical midi data is semantically equal to the physical midi data but presents the data in a natural way to the application using “native” data types.

1.2

Validation of data based on the Midi Device Schema.

The Schema itself contains all validation rules and can be used to validate incoming logical, and if desired, physical data streams.

2.0

An addressable index into the schema must be provided.

- no solution yet.

2.0.1

Repeatable records must be supported (addressable index)

- no solution yet. The data record (complex) type will be used as boundary for the repeatable records.

2.1

Multiple field transformations must be supported.

The (data) Converter

2.1.1

Specify a dummy field. Can be used to align a data record on a specific boundary.

The midiNull data type provides a means to specify dummy fields.

2.1.2

Allow a logical value to be a composite of more physical values. It must be possible not to use all the bits in the physical values (for instance: 3 physical bytes filled with (LS) nibbles make up a logical 12 bit value).

A custom data type can use a union to produce these results. Also refer to the (complex) data record.

2.1.3

A textual string field value. Multiple characters represented as one logical value.

The midiString data type forms the basis for reading textual string data.

2.1.4

Allow a Boolean field value for any bit.

The midiBit0 – midiBit7 data types will allow a Boolean value for any bit.

2.1.5

Constrain a field value.

Any custom data type can specify constraints using Xml Facets.

 

Merged value. One physical byte that contains multiple logical field values spanning one or more bits each.

A data “carry” will provide this functionality during a To-Logical or To-Physical data transformation.

 

Enumerations. A sequence of named values.

Any custom data type can use the Xml Enumeration Facet to fulfill this requirement. Use the midi:name attribute on each enumeration facet to name the value.

2.2

Multiple data record (=multiple fields) transformations must be supported.

The GroupConverter

2.2.1

Bit shift algorithms.

The extensibility of the GroupConverter provides in this need. Some (unknown which) bit shift algorithms will be implemented.

2.2.2

Checksum calculations.

Typical checksum algorithms are provided as a group converter (data record base type). Custom checksum calculations can be made by declaring a custom record type and extending the group converter.

3.0

The physical-to-logical and visa versa transformation of data should be extensible.

A factory pattern is used to create the data converters. All types from the Base Type System are provided. Custom types can be added.

 

 

 

 

Because the data converter framework is extensible, any requirement not listed here can probably be implemented through a custom converter or group converter.

Schema

The term Schema represents a description of a digital data structure; midi messages in this case. The proposed solution sets down a type system as a basis for describing the device’s System Exclusive implementation. The schema describes all aspects of a System Exclusive message; from message exchange patterns to field data types.

 

The choice was made to express the Midi Device Schema as Xml Schema xml. Because almost all of the constructs needed are readily available in Xml Schema and the parsing technology is free.

 

There are some liberal interpretations of Xml Schema in the Midi Device Schema language.

 

Xml Schema

Midi Device Schema

simpleType

A data type to be used in a field

complexType

A logical field record; grouping of fields.

element

A logical field declaration

union

Mask, shift and bitwise-OR a data byte

fixed

A fixed midi protocol or device specific value.

abstract

Used on a field type to indicate a place holder to be overridden by derived schema types.

nillable

Specified on a field to make that field optional.

facet

A constraint on the value of the field.

annotation

Used for documentation.

midi:name

A custom name attribute used to name facets (enumerations).

 

Declarations

The following sections show how to declare Midi Device Schema constructs in Xml Schema.

 

The examples use “mt” as the namespace alias for the midi base type’s namespace and the “xs” namespace for the Xml Schema elements.

 

Midi Device Schema declaration

A Midi Device Schema declaration is exactly the same as an Xml Schema declaration. The following example illustrates a basic schema declaration.

<?xml version="1.0" encoding="utf-8"?>

<xs:schema targetNamespace="[name of schema]" xmlns:xs="http://www.w3.org/2001/XMLSchema">

 

</xs:schema>

Note that the targetNamespace is used to identify the Midi Device Schema. This name should be chosen such that it uniquely identifies the midi device and its System Exclusive implementation. Schemas with the same targetNamespace are considered to be the same schemas.

 

Including another Schema

Using the standard Xml Schema includes other schemas (the base types for instance) can be included into a custom schema.

 

<xs:schema targetNamespace="[name of schema]" xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:import namespace="[schema]" schemaLocation="MySchema.xsd" />

 

Declaring a data type

Data types are the basis for converting physical midi data into logical midi data and visa versa. The data converters are created based on these data types.

 

This example shows the mother of all data types, the midiByte.

<xs:simpleType name="midiByte">

  <xs:restriction base="xs:unsignedByte" />

</xs:simpleType>

 

Note that typically custom data types are derived from existing data types. See also Inheriting a data type.

 

Declaring a data record

A data record is a collection (sequence) of fields that logically make up a unit. How a device’s System Exclusive data fields are grouped into records is a choice of the schema author.

 

This example shows the template for a System Exclusive message. Note the begin- and end-markers of the midi SysEx and an abstract type as a place holder for the message body content.

<xs:complexType name="midiSysExMsg">

  <xs:sequence>

    <xs:element name="SOX" type="mt:midiSysEx" fixed="240" />

    <xs:element name="Body" type="midiSysExMsgAbstractBody" />

    <xs:element name="EOX" type="mt:midiSysEx" fixed="247" />

  </xs:sequence>

</xs:complexType>

 

Declaring a field

A field represents a logical value in a SysEx message and is always declared as part of a data record.

 

This example show the declaration of the Universal SysEx Id field (using a contrained data type).

<xs:complexType name="…">

  <xs:sequence>

    <xs:element name="UniversalId" type="mt:midiUniversalSysExID" />

   

  </xs:sequence>

</xs:complexType>

 

Note that a field’s type can be either a data type (representing a concrete value) or another record type (representing a sub-record of fields).

 

Inheriting a data type

Currently only restriction-inheritance is supported for data types.

 

The example shows the declaration of the midiSysExData data type that is constrained to bits 0-6 (not shown).

<xs:simpleType name="midiSysExData">

    <xs:restriction base="midiByte">

     

    </xs:restriction>

  </xs:simpleType>

Declaring custom data types is a matter of picking the most appropriate base type and optionally adding constraints.

 

[Implementation:] In order to overrule data converter functionality one could declare a new data type and inject their custom data converter for that data type by providing a converter factory.

 

Inheriting a data record

A data record (complex type) can be inherited in order to override abstract fields in the base type. See also Overriding a field.

 

The following example shows the declaration of a universal SysEx message based on the midiSysExMsg template presented earlier.

<xs:complexType name="midiUniversalSysExMsg">

    <xs:complexContent>

      <xs:extension base="midiSysExMsg">

        <xs:sequence>

         

        </xs:sequence>

      </xs:extension>

    </xs:complexContent>

  </xs:complexType>

Note that deriving data records (complex types) is by extension, not restriction as with data types.

 

Overriding a field

In order to override an existing (abstract) field on a base type the names of the field should match exactly.

 

Note that only fields whose type is another data record type can be overridden.

In the following example the Body field of the midiSysExMsg base type is overridden with a new type. The midiUniversalSysExBody is a data record type that declares all the fields for a universal System Exclusive message.

<xs:complexType name="midiUniversalSysExMsg">

    <xs:complexContent>

      <xs:extension base="midiSysExMsg">

        <xs:sequence>

          <xs:element name="Body" type="midiUniversalSysExBody" />

        </xs:sequence>

      </xs:extension>

    </xs:complexContent>

  </xs:complexType>

 

Field overrides are name-based and the type of the field in the base does not have to be abstract in order to be overridden. So whenever a derived data record specifies a field who’s name is present in its base record, it will override that field with the (data) type specified. The field (data) type in its base will not be used, unless the new field (data) type derives from it.

 

Note that it is permitted to specify fields with the same name in different (complex) types for a Midi Device Schema (as is the case when overriding a field). Xml Schema parsers will generate an error for this condition which can be ignored.

 

Declare a Composite data type

 

A new custom data type that specifies a union of other data types will provide this behavior.

 

Data type constraints

An Xml Schema facet can be used to constrain the valid value range of a data type.

 

The following example shows the constraints placed on a data byte in a SysEx message.

<xs:simpleType name="midiSysExData">

    <xs:restriction base="midiByte">

      <xs:minInclusive value="0" />

      <xs:maxInclusive value="127" />

    </xs:restriction>

  </xs:simpleType>

 

The following xml facets are supported:

Xml Facet

Description

minInclusive

Specifies the minimal value for the data type. The value specified lies inside the valid range.

maxInclusive

Specifies the maximal value for the data type. The value specified lies inside the valid range.

minExclusive

Specifies the minimal value for the data type. The value specified lies outside the valid range.

maxExclusive

Specifies the maximal value for the data type. The value specified lies outside the valid range.

enumeration

Specifies an exact value that is valid for the data type. Multiple enumeration facets can be used fro one data type.

length

Specifies the fixed length of a data type. Only valid for data types deriving from midiComposite.

 

 

Constant values

There are two ways to declare constant values.

 

  1. Constraint a custom data type
    Constraint a custom data type with one enumeration facet that specifies the constant value. Use this data type for all fields that require the constant value.
  2. Specify the fixed attribute for a field.
    For each field that requires the constant value specify the fixed attribute and the constant value.

 

The following example shows the fixed values for the SysEx begin- and end-markers in a midi SysEx message.

<xs:complexType name="midiSysExMsg">

  <xs:sequence>

    <xs:element name="SOX" type="mt:midiSysEx" fixed="240" />

    <xs:element name="Body" type="midiSysExMsgAbstractBody" />

    <xs:element name="EOX" type="mt:midiSysEx" fixed="247" />

  </xs:sequence>

</xs:complexType>

 

Documenting a schema

You can use the standard Xml Schema annotation tags to document the Midi Device Schema declarations.

 

<xs:schema targetNamespace="…" xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:annotation>

    <xs:documentation>The midiTypes schema represents the base types used in the midi device schema framework.</xs:documentation>

  </xs:annotation>

  <xs:simpleType name="midiByte">

    <xs:annotation>

      <xs:documentation>midiByte is the base type for all (simple) midi data types. It represents an unsigned 8-bit byte.</xs:documentation>

    </xs:annotation>

    <xs:restriction base="xs:unsignedByte" />

  </xs:simpleType>

 

</xs:schema>

 

The Type System

[Note: The type system is still under development!]

 

The Midi Device Schema type system is designed to allow a schema author to express the logical midi System Exclusive fields of a device. All types inherit from the midiByte data type which represents an unsigned byte (8-bits).

 

The following figure displays the type hierarchy for the (simple) data types declared in the type system. The lines represent inheritance and the arrows point to the derived (more specific) type.

 

 

 

midiByte

Represent the base data type for all other types but is not declared abstract and thus can be used as any data type. The value represents the full 8 bits of an unsigned byte.

 

midiBit0-midiBit7

The midiBit0 till midiBit7 data types each represent one bit of the 8-bit byte. midiBit0 represents the least significant bit and midiBit7 represents the most significant bit.

 

midiNull

The midiNull data type can be used for fillers or dummy fields. The value defaults to 0 (zero) but can be set at a per field basis using the fixed attribute on the field declaration. Fields that use this data type are not visible at a logical level. When logical data is converted to physical midi data, these fields are inserted automatically at their designated positions within the record.

 

midiSysExData

The midiSysExData represents a data byte inside a midi System Exclusive message. This data type enforces that bit7 (most significant bit) is always clear (not set) to comply with the midi specifications for System Exclusive messages.

 

midiNibble

The midiNibble data type is abstract and cannot be used in a field declaration. The data type is a logical indication that the data is 4-bits in length (a nibble) and is mainly used by the nibble-data converter implementation.

 

midiLSN

The midiLSN represents the Least Significant Nibble (LSN) of a data byte (lower 4 bits).

 

midiMSN

The midiMSN represents the Most Significant Nibble (MSN) of a data byte (upper 4 bits).

 

Note that a contradiction is hidden in this data type: The midiMSN derives from midiSysExData (via midiNibble). That type ensures that bit7 is always cleared. This means that bit7 of a midiMSN field is also always cleared.

 

midiComposite

The midiComposite is an abstract type that is used as a base for logical values that span multiple physical bytes.

 

This type is not intended to be derived by schema authors.

 

midiString

The midiString (derives from midiComposite) represent multiple physical data bytes that make up one logical text string.

 

In order to use this in a schema declaration one should derive a custom data type and specify the length facet for that custom type. The string-data converter will use the specified length to read the text string.

 

Record Base Types

The Type System also contains (complex) record types (field structures). Checksum calculations for instance can be implemented this way. These types are still under development and too volatile to document here. These will be added as soon as the design is somewhat more stable.

 

The current functionality for these (complex) record base types include:

 

 

 

Implementation details

This section delves into implementation details for these specifications. Some assumptions exist on how the implementation should handle certain events. This description follows the base architecture provided at the beginning of this document.

Schema Object Model

The schema object model provides an in-memory representation of a Midi Device Schema declared in an .xsd file (or other resource).

 

The most important feature of this component is the ability of interpreting the xml in the schema and compiling this into a fully linked object model.

 

These operations are performed during the compile phase:

 

 

The following classes could be expected to be in a Schema Object Model:

Class

Description

SchemaManager

An entry point for caching and loading schemas. Is used during schema compilation to resolve imported types.

Schema

The root object for a specific schema. Created based on an .xsd file (or resource). Contains all declarations made in the Midi Device Schema.

RecordType

A representation of a data record containing fields.

DataType

A description of a (custom) data type.

Field

A description of a field. Including Declaring RecordType and Constraints.

Constraint

A description of a type constraint.

For each class, except the SchemaManager, a collection or list class should also be available.

 

(Refer to the Microsoft .NET System.Xml.Schema namespace for inspiration.)

 

Flattened Field List

A record type consists of fields. Each field can either be a value (data type) or a sub-record (record type). This produces a hierarchy of record types where data fields are leaves in the tree. A flattened field list is a list of all fields effectively in a record type. This list will only list data field (with data types) and is basically the flattened structure of the system exclusive data.

Data Object Model

A typical scenario for the Data Object Model would be to transform data for a specific midi device. The Midi Device Schema (class) from the SOM is interpreted and the data converter factories are called to create data or group converters for all data and record types. Once the converters are in-place the DOM can start to receive transformation requests to transform from physical midi data to logical midi data or visa versa.

 

It stands to reason that the DOM supplies some sort of caching mechanism for the converters (they are stateless) and a class to expose the data transformation services based on a specific schema.

Data Converters

The hierarchy of data records described in the Schema Object Model section also reflects back onto the data converters. Data converters are created through registered converter factories. The factory supports two create methods; one for creating a data converter for a data type and the other for creating a group-converter for a record type. Group converters are containers for other data and group converters. For each field in the data record a data converter is added to the parent group converter. Group converters are responsible for forwarding the to-logical and to-physical calls to their children. This gives them the unique opportunity to intercept the to-logical or to-physical data transformation calls and perhaps wrap the physical or logical streams.

 

Data converter logic is targeted on the data type of a logical field definition. The most basic ‘conversion’ just copies the data values during conversion. For instance, the data converter for midiString will read or write multiple bytes during a transformation call.

 

Group converter logic is targeted on the data record type; a group of fields. The default group converter merely forwards the data transformation calls to its children. The group converter is capable of gathering context information during a data transformation. Think of calculating checksums by intercepting the values each child data converter writes to the physical stream and applying some sort of algorithm. Group converters do generally not show up in the logical data. The mainly play a role handling the physical midi stream.

 

Each converter implements its own logic to transform physical data to logical (ToLogical) or visa versa (ToPhysical). Important is to keep the converter implementations stateless. This means they cannot have any instance members – that need to change after initialization. Also, because converters are cached they need to be thread-safe. One instance of a converter can be used (simultaneously) in multiple data transformations.

Data Transformation

Data transformation starts when the application code calls into the DOM and presents a Midi Device Schema root data record type, a physical data stream and a logical data stream.

 

The root data record type represents the Midi SysEx message that needs to be read or sent. The flattened field list of this data record is enumerated during the data transformation. The physical stream is either the source (in a to-logical transformation) or the target (in a to-physical transformation) and represents the physical midi data as it would appear on the wire. The logical data stream is an application implementation of a (set of) interface(s). It is either asked to provide logical values for a specific field (in a to-physical transformation) or to receive logical values for a specific field (in a to-logical transformation).

 

During a data transformation the Carry provides a way to build up a physical byte of data during calls to multiple data converters. When the Carry is full, or when a value is written that requires a new byte, the Carry byte is flushed. The carry provides the mechanism to declare multiple logical fields on one physical byte. The Carry is transparent for the converter implementation but can be accessed explicitly if desired (and with that comes a responsibility not to screw it up).

 

Resources

 

Xml Schema:

http://www.w3.org/XML/Schema

http://www.w3schools.com/schema/default.asp

http://msdn.microsoft.com/XML/BuildingXML/XMLSchemas/default.aspx

 

Microsoft .NET:

http://msdn2.microsoft.com/en-us/library/system.xml.schema.aspx

 

Midi:

http://www.midi.org/about-midi/specinfo.shtml

http://www.borg.com/~jglatt/tech/miditech.htm