Conceptually represented example Annodex file, time increasing from left to right 

Context in source publication

Context 1

... is the format in which media with interspersed CMML markup is exchanged. Analogous to a normal Web server offering a collection of HTML pages to clients, an Annodex server offers a collection of Annodex resources. After a Web client has issued a URI request for an Annodex resource, the Web server delivers the Annodex resource, or an appropriate subpart of it according to the URI query parameters. Annodex resources conceptually consist of one or more media streams and one CMML annotation stream, interleaved in a temporally synchronized way. The annotation stream may contain several sets of clips that provide alternative markup tracks for the Annodex resource. This is implemented in CMML through a track attribute of the clip tag. The media streams may be complementary, such as an audio track with a video track, or alternative, such as two speech tracks in different languages. Figure 2 shows a conceptual representation of an example Annodex resource with three media tracks (light colored bars) and two annotation tracks (darker clips) with a header describing the complete resource (dark bar at the start). The Annodex format enables encapsulation of any type of streamable time-continuous data and is thus independent of a media compression format. It is basically a bitstream consisting of continuous media data interspersed with the structured XML markup of the CMML file. This is performed by merging the clip tags time-synchronously with the time-continuous bitstreams on authoring an Annodex bitstream. The clip tags are regarded as state changes in this respect and are valid from the time that they appear in the bitstream until another clip tag replaces them. If there is no clip that directly replaces a previous one, an empty clip tag is inserted that simply marks the end of the previous clip tag. Thus, Annodex is designed to be used as both a persistent file format and a streaming format. Figure 3 shows an example of the creation of a bitstream of an Annodexed media resource. Conceptually, the media bitstreams and the annotation bitstreams share a common timeline. When encapsulated into one binary bitstream, these data have to be flattened (serialized). CMML is designed for serialization through multiplexing. The figure shows roughly how this is performed. There are several advantages to having an integrated bitstream that includes the annotations in a time-synchronous manner with the media data. Firstly, all the information required is contained within one resource that can be distributed more easily. Also, many synchronization problems that occur with other media markup formats such as SMIL [31] are inherently solved. Also, when extracting temporal intervals from the resource for reuse, the metainformation is included in the media data, which enables one, e.g., to retain the copyright information of a clip over the whole lifetime of a reused clip. Last but not least, having a flat integrated format solves the problem of making the Annodex resource streamable. To perform the encapsulation, a specific bitstream format was required. As stated, an Annodex bitstream consists of XML markup in the annotation bitstream interleaved with the related media frames of the media bitstreams into a sin- gle bitstream. It is not possible to use straight XML as encapsulation because XML cannot enclose binary data unless encoded as Unicode, which would introduce too much overhead. Therefore, an encapsulation format that could handle binary bitstreams and textual frames was required. The following list gives a summary of the requirements for the Annodex format bitstream: • Framing for binary time-continuous data and XML. • Temporal synchronization between XML and time-continuous media bitstreams. • Temporal resynchronization after parsing error. • Detection of corruption. • Seeking landmarks for direct random access. • Streaming capability (i.e., the information required to parse and decode a bitstream part is available at the time at which the bitstream part is reached and does not come, e.g., at the end of the stream). • Small overhead. • Simple interleaving format with a track paradigm. We selected Xiph.Org’s [36] Ogg encapsulation format version 0 [20] as the encapsulation format for Annodex bitstreams as it meets all the requirements, has proven reliable and stable, and is an open IETF (Internet Engineering Task Force, ) standard [20]. Hier- archical formats like MPEG-4 or QuickTime were deemed less suitable as they are hierarchical file formats and therefore could not easily provide for streamable, time-accurate interleaving of multiple media and annotation tracks. To author Annodexed media, we must distinguish between files and live streams. The advantage of the former is that a file can be uploaded from the computer’s file system and annotated in a conventional authoring application. In contrast, the markup of a live Internet stream by its very nature has to be done on the fly. Annodex media files may be created in a traditional authoring application (e.g., iMovie or Adobe Premiere may easily support Annodex in the future) or through the use of CMML transcoded from metainformation collected in databases. The authoring application should support the creation of: • Structured and unstructured annotations, • Keyframe references, • Anchor points, and • URI links for media clips. Live Annodexed media streams must be created by merging clip tags with the live digital media stream. A merger application, similar to that described in Fig. 3, in- serts clip tags into the live stream at any point in time under the control of a user, e.g., by selecting a previously prepared clip tag from a list. It is expected that extending existing graphical video editing applications such as Apple’s iMovie or Adobe’s Premiere to author Annodex will be a simple task. Most already provide for specific markup to be ...