Much of the time spend on this project was dedicated to finding a solution to system specification and stabilising a high level design. The design has undergone many iterations and revisions and in this section, I present the final design with a minimum number of alternatives which were considered.
The final state of the high level design is shown in figure 4.1 An attempt has been made to build fault-tolerance into the infrastructure, by returning an error flag with each function call. This means that errors are percolated throughout the system, up to the user level.
Figure 5.1 - The high level dataflow diagram
Events are specified in files, and are handled by the system in a generic event language (GEL). GEL events are mapped onto device-specific messages at perform-time. The generic event language is built up by considering the capabilities of each device, and assigning universal keywords to perform that action. This allows the portability of files.
GEL keywords are always of the form:
keyword_name(p1 : t1, .., pn : tn)
For example, a musical device can play a note. This action could be captured by a "note" keyword of the form:
note(pitch : string, velocity : byte, duration : byte)
In reality, this generic event may be mapped onto two MIDI messages:
[&8n][note_number : byte][velocity : byte] - the note on message
[&9n][note_number : byte][velocity : byte] - the note off message
(Where n is the MIDI channel being used.)
MIDI is an example of one control language being used, but GEL events may be mapped onto different control protocols, which is why the most general form of the event must be sought.
Some GEL commands might be private to the system, and not seen by the user. These would include things like messages sent to a routing matrix (see 4.9.3), or a message to an effects device. Messages sent to these devices would be based on the changes to blocks, or to the structure of a file’s dataflow diagram, and would be sent automatically by the file processor. The user would never state these particular events explicitly.
Conversely, other GEL messages may never actually be sent to a device. These would include messages sent to a block in the file’s dataflow diagram which is an GEL modifier, such as transpose, velocity fix, time offset, etc. These events would be understood by a component of the file processor rather than a virtual device.
Generic event keywords must be standardised for all functions performed on each of the file dataflow blocks. This may lead to overloading keywords, eg. on() means ‘light on’ or ‘note on’, and value(7) may refer to ‘transpose’ value or ‘velocity fix’ value. This doesn’t pose any problems, however, because the keyword will always make sense when used in the context of it’s intended device.
See appendix 8.1 for a list of example GEL keywords.
The sequencer consists of various interface tools described further in 5.1. Actions within the interface are converted into messages which are sent to other objects.
The sequencer is responsible for displaying it’s local copy of the schedule in the desired format. The sequencer must also allow the user to create or edit files. This means converting file headers into dataflow diagrams, allowing the user to edit them, and then converting them back into ASCII files. The user may wish to monitor files independently of the main output. This would be performed by updating a files header, to route audio/visual data to a local monitoring device. When the user wishes to play the file using the main output, the header is reverted to it’s original form.
In a distributed environment, one sequencer may be flagged as the ‘master’ sequencer. This sequencer would control stop, start, changes to the tempo etc., where other sequencers would not be able to. The sequencer would be responsible for loading and saving compositions, that is - the state of the schedule, a copy of all concerned files and further necessary data. The user can transfer their work from site to site by means of compositions.
error(file, error)
Pops up an error box which informs the user of an error in
the system or in a particular file.
set_schedule(schedule)
Updates the local copy of the schedule in the sequencer.
This change must be reflected in the interface views.
set_time(time)
Updates the local copy of the global clock in the
sequencer. This change must be reflected in the interface views
(ie. by advancing the cueing window, updating the displayed clock
etc.)
Schedule, which is not necessarily the most up to date version.
Clock, which is not necessarily the most up to date version. The sequencer is responsible for keeping it’s local clock sync with the global clock.
A file dataflow definition, if the user is currently editing a file. This definition may be associated with a history of changes, to allow the user to undo changes, as well as switching the output devices to main/monitor.
Messages sent to the scheduler would include:
play(filename, start_time) :
(file_id, match, error)
This tells the scheduler to play a certain filename at a
certain time. This function returns a file ID, details of how
well resources have been matched, and an error status flag (eg.
file does not exist).
remove(file_id) : error
Removes a file from the schedule.
update(file_id, header) :
(match, error)
This tells the scheduler to re-schedule the file, using this
header instead of using the header of the stored file. This is
used when making changes to the file’s dataflow diagram -
ie. adding, removing or moving blocks.
get_devices(start_time,
end_time) : set{device_id}
This call to the sequencer returns a set of devices which
are available from start_time until end_time. This is used when
the user wishes to know which devices are available if a
particular device match cannot be made automatically.
The sequencer sends the following message to the file processor:
update(file_id, block_id,
gel_event) : error
This tells the file processor to make changes to a
block’s properties in the dataflow diagram of a file in the
file processor. This is used when eg. the user is altering the
transpose value of a file in real-time.
The load and save messages are sent to the filestore to allow the user to create/edit and save a file:
save(filename, textfile) : error
load(filename) : (textfile, error)
The get_time() message is sent to the global clock to keep the sequencer’s clock in time.
The scheduler must hold the global file schedule and the global device (locks) schedule. Virtual devices correspond to a physical piece of hardware, and may be locked. Private virtual devices (eg. routing matrixes) and GEL modifiers are used by multiple files simultaneously and are therefore never locked.
The scheduler must accept a location-specific filename and a start time. The scheduler loads the file header, and parses it to ensure the dataflow structure is valid. The scheduler must then attempt to match each desired device in the header to a device id. The scheduler stores this new header, along with the filename, and passes it to the file processor when the file is ready to be played.
The scheduler must also constantly check the schedule, and terminate the playing of any files which have ‘ended’. This means terminating a file processor, which would otherwise be idling, or terminating a file processor which is actively playing a loop (see 3.1).
play(filename, start_time) : (file_id, match, error)
This ‘plays’ a certain filename at a certain time. Given a location-specific filename, the scheduler requests the file header from the filestore and parses it to ensure validity. The scheduler then attempts to match each desired device class in the header to a device_id, or match a GEL modifier to a library function. The virtual device_id, or GEL modifier name is stored in the header. The matched block may be:
The match variable returned may thus indicate that;
The error variable returned may indicate that;
The scheduler must then store the new header, along with the original filename until the concerned file is about to be played. The scheduler then spawns a new file processor, and sends it the new file header, the start time and the filename.
remove(file_id) : error
This function must remove a file from the schedule, free up all the devices allocated to it, and instruct the file processor stop playing the file and terminate.
update(file_id, header) : (match, error)
This tells the scheduler to re-schedule the file, using this header instead of using the header of the stored file. This is used when making changes to the file’s dataflow diagram - ie. adding, removing or moving blocks. The scheduler must perform a remove on the current file, and then perform a function similar to play, except the scheduler is passed the file header from the sequencer, not the filestore. The unique file ID is preserved.
get_devices(start_time, end_time, input_datatype, output_datatype) : set{device_id}
This function returns the set of device classes which are available from start_time until end_time. This is used when a device used by a file cannot be completely matched, and the user wants to know which device classes are available that will suffice.
get_modifiers() : set{modifers}
This function returns the set of GEL modifiers which are present in the system (see 4.6). GEL modifiers are stored as library functions in the filestore, and users may create their own. This means that from site to site, GEL modifiers may vary. This functions allows the user to see which GEL modifiers are available. This function could be provided by the device registry, which actually holds these details. However, the get_devices request must be handled by the scheduler, and putting the get_modifiers function here maintains this consistency.
The global file schedule is held as:
file_schedule= list{(file_id, match, start_time, end_time, device_specific_header, filename)}
The global device schedule is held as:
device_schedule= list{(device_id, start_time, end_time, owner : file_id)}
The scheduler sends a match(device_class) message to the device registry, and is returned a set{device_id}, representing all instances of the desired device class.
The header(filename) message is sent to the filestore, and returns a file’s header, or an error message.
The load(device_specific_header, filename, start_time) message is sent to the file processor who will process the file’s dataflow diagram and send GEL messages to all concerned virtual devices.
Files consist of a header, zero or more event lists, and are held in the filestore.
The header defines a file’s dataflow structure, as shown in figure 3.1. The header must contain all information necessary to connect blocks together in this way, and store parameter settings for each block. This, in effect, is a ‘program’ for the file processor to run.
Each block in the dataflow structure is held as a:
block(block_id)= (input_datatype, output_datatype, list{(class_name, list(gel_event))})
It consists of:
The dataflow structure is a non-cyclic, directed graph. It can therefore be recorded in a file as:
dataflow= set{(source : block_id, set{target : block_id})}
Multiple files may be grouped into a single, composite file. This is performed by merging the files. A dataflow diagram need not necessarily be a connected graph - ie. more than one distinct dataflow path may be described by the same file header. Grouping multiple simply files means merging the two files, and re-enumerating each block and the graph description.
See appendix 8.2 for examples of files.
Zero or more event lists are recorded in the file. Event lists are ordered on time, in ascending order. Each event list has an identifier, to tie it to a block in the dataflow diagram. Event lists are of the form:
event_list(block_id)= (name, list{(time, gel_event)})
time is of the form bar:beat:tick, and may be negative. This is used when a musical part has a brief introduction, such as a drum roll, but must be seen to ‘begin’ at time 0, that is 1:1:0. The scheduler snaps time 0 to the nearest bar.
name is a name given to the event list by the user, eg. "striking chords". This name appears as a label on the dataflow diagram.
If the event list is looped, then a (end_time, loop(begin_time)) event must be present. It would be interpreted by the file processor as the contents of the event list, starting at time begin would be pasted at time end. (See 3.1).
GEL modifiers are functions which accept GEL keywords, and some input parameters, and return an updated GEL keyword. In this design, they are held as executable software objects in a library - perhaps in the filestore, and are used by the file processor at run-time. They have a known interface, and may be used by any component of the system - ie. they are never locked by the scheduler.
GEL modifiers can be thought of as functions which accept (time, gel_event) events and return (time’, gel_event’) events.
The device registry is a database of all virtual devices and GEL modifiers attached to the system. A virtual device is any hardware entity which can be controlled individually. That means that a 16-channel MIDI keyboard can be thought of as 16 separate virtual devices. A GEL modifier, although not a device as such, also comes under this category. Any block on a file’s dataflow structure (except input & event lists) has a corresponding entry in the device registry.
Furthermore, the device registry must contain the set of universal, ‘generic’ devices. (See 2.1.3)
devices= set<(device_id, computer, device_class, input_datatype, output_datatype)>
For each device, the device registry holds:
match_class(device_class) : set{device_id}
The match_class function accepts the name of a device class, and checks to see if there are any instances of that device class present on the system. If there are, match returns a set containing the device ID’s of all matching devices. If not, match returns the empty set.
match_datatype(input_datatype, output_datatype) : set<device_id>
match_datatype accepts input and output datatypes, and returns a set of device ID’s which support the specified datatypes.
A file processor is a complex, CPU-intensive component of the system. A file processor exists for each file which is being played.
load(filename, device_specific_header, start_time) : error
This function creates a new file processor and initialises it with a device-specific file dataflow structure, and a start time. The file processor loads the original file, and discards the header.
According to the device-specific header, the file processor sends initialisation messages to all virtual devices, including any routing matrixes. This sets up each device (eg. sets a MIDI instrument to certain sound, a routing matrix to a certain configuration), and sets the audio or video’s path to it’s final destination.
The file processor must then examine each event list, and wait for an event which is about to be played by calculating the event time, start time and the current time. ‘About to be played’ means that the event is due to be played within some look-ahead time, eg. 1 beat. This time used as a buffer when using a network which induces delays. Events previous to the current time are discarded. If the looped property for the event list is true, then the file processor cycles indefinitely through the event list. A ‘terminating’ event in the event list, loop(time) would instruct the file processor to roll back to time.
The single event, along with it’s time code are then ‘fed into’ the corresponding block on the dataflow description, and percolated throughout it’s structure. This means that the event must be acted upon by each GEL modifier along it’s path. The file processor would call the modifier function with the current GEL event, and corresponding input parameters. This would happen with each modifier in the path, until the event is sent to a virtual output device (a device where input_datatype=gel and output_datatype<>gel). The signal requires no further processing on the part of the file processor after it has passed this point, as this is taken care of by the routing matrix which is configured when the file is initialised.
When an event is sent to a virtual device, it is buffered there, along with it’s time code. The event, at this point, is not actually time-critical, ie. there is still a look-ahead delay of maybe 1 beat until is it played. The virtual device outputs the event on time with the global clock, from which point on, the event is in real-time, and is assumed to be converted into sound/graphics instantaneously. This means that the virtual device is always located on computer supporting the real device - the network layer has already been traversed, and does not cause any delay when the event is fired off in real time.
As mentioned, each event in an event list is released when it is about to be played. If the entire event list was processed and sent to the virtual devices as soon as the file was initiated, then any real-time modifications to blocks on the dataflow structure would not be reflected. To illustrate, imagine that a user has initiated a file which plays a musical part:
[event list] -> [transpose] -> [virtual device] -> [speaker]
When the file starts playing, the user notices that it is out of tune, and alters the transpose value. If the event list had already been sent to the virtual device, altering the transpose parameter would have no effect, as the note events would already be waiting, with the wrong note value, in the virtual device buffer. If, however, the events were streamed out of the event list eg. 1 beat before they were played, then the transpose would be guaranteed to effect notes no later 1 beat. This ‘1 beat’ look-ahead value, would be set according to the speed of the system, and minimised to allow user input to take near-immediate effect.
terminate()
This function stops processing the current file, and destroys the current file processor. It is used when a user deletes a file which is currently playing.
play(file, block_id, gel_event)
This function injects a single event into a block on the file processor. The event is time stamped as ‘now’ and is processed in the same way events from an event list are. This is used when accepting input in real-time.
A virtual device is a process running a computer. The real, physical device is connected to the computer the virtual device is running on. Virtual devices can be thought of as a library of implementations, which are spawned when the system is booted. The virtual device knows the physical address of the device, and knows how to communicate with that device. A virtual device supports a subset of GEL keywords.
A virtual device can be passed a GEL keyword and a time, and will insert them into an ordered buffer. It waits in a loop, comparing the current time with the head(buffer) time. When an event is due, it converts head(buffer) into a device-specific messages, (or otherwise acts upon it). A virtual device, must therefore have a well-defined interface, which specifies which GEL keywords it supports (and implements). If the virtual device cannot support the message, then it ignores it. A virtual device makes use of a local clock which is kept in time with the global clock.
keyword_name(p1 : t1, .., pn : tn) : error
Eg: note(pitch, velocity, duration)
A function of the virtual device is called in this way. keyword_name is the name of a service the virtual device provides, and can be equated to a GEL keyword. A number of functions may exist, each with zero or more input parameters.
property(value) : error
Eg: patch(name), volume(value)
property is the name of a property the virtual device supports and remembers. It is implemented in exactly the same way a keyword is.
get_property() : (value, error)
Eg: get_patch():name, get_volume():value
Opposite of property(), this function returns the property specified. This keyword would never actually appear in event lists, but may be used by the user for information/debugging.
A virtual device may also be a non-standard device, such as a graphics program. The interfaces of different graphics programs vary a great deal and it would be impractical to consider these function names as GEL keywords.
It would be helpful, however, if programs such as these provided a note(pitch, value, velocity) (or another universal format) because they would be able to interpret standard musical events and would be useable in more situations.
buffer= list{(time, GEL_event)}
This is a list of events waiting to be streamed. It is ordered on time in ascending order. In reality, it is a list of calls the virtual device supports, which are called when time >= global_clock.
The virtual device must hold details of it’s own properties as a data structure. These properties are private to the virtual device, and are accessible only through the interface. Properties will vary in type and number from device to device.
An audio or visual routing matrix (see 2.1.1) is a virtual device which provides a connect(device_id, device_id)service.
A physical routing matrix simply connects an input socket to an output socket when receiving a control signal to do so. The virtual routing matrix must therefore know which device_id’s are located at which input/output sockets. The virtual router must then express the connect(input socket, output socket) message into eg. MIDI and send it to the routing device.
In theory, conflicts should never occur here, because only the file processor sends messages to the virtual router. The file processor will always have got permission from the scheduler to use the concerned devices.
Virtual input devices support a send_to(file_id, block_id) keyword. A virtual input devices waits in a loop, listening for control signals from real devices which match it’s address. These are converted into GEL data and are send to file processor file_id, with the parameter block_id. Some virtual input devices may support a name(name:string) keyword, which would set a text display located on the device. This would be useful eg. when the user was using a bank of MIDI sliders, each, of which, had an associated LCD display.
Another class of virtual device is a MIDI clock. This device doesn’t correspond to a physical MIDI device, but does produce MIDI clock data on the MIDI port it is acting upon. This is used to provide conventional MIDI timing data.
Generic devices are output devices such as speakers, graphics windows which must be supported by all implementations of the system to allow portability. In reality, these are just another set of device classes, which map onto specific instances of speakers etc.
However, a universal naming scheme would need to be put forwards. One possibility here would be to use the names of chess pieces. This would mean that the set of speakers are recorded as eg. "King Speakers" for the principle speaker pair, "Left Bishop", "Right Bishop", "Pawn 1", "Pawn 2" etc. This approach would also be suitable for graphics windows.
A naming scheme for lights would be slightly more difficult, due to the variation in number and type. Lights may perhaps be called, "Red Circuit", "Blue Circuit", "Main Spotlight".
The global clock holds the current time in bar:beat:tick format. It is responsible for keeping the time updated, according to the tempo it has been set with.
set_clock(time)
get_clock() : time
set_tempo(tempo)
get_tempo() : tempo
time : (bar : integer, beat : byte, tick : integer)
tempo : real
This document composed by Adam Buckley (adambuckley@bigfoot.com), last edited on 16-May-2002.