Merging of the frames of several media bitstreams with a structured CMML file into an Annodexed bitstream 

Contexts in source publication

Context 1

... information of a clip over the whole lifetime of a reused clip. Last but not least, having a flat integrated format solves the problem of making the Annodex resource streamable. To perform the encapsulation, a specific bitstream format was required. As stated, an Annodex bitstream consists of XML markup in the annotation bitstream interleaved with the related media frames of the media bitstreams into a sin- gle bitstream. It is not possible to use straight XML as encapsulation because XML cannot enclose binary data unless encoded as Unicode, which would introduce too much overhead. Therefore, an encapsulation format that could handle binary bitstreams and textual frames was required. The following list gives a summary of the requirements for the Annodex format bitstream: • Framing for binary time-continuous data and XML. • Temporal synchronization between XML and time-continuous media bitstreams. • Temporal resynchronization after parsing error. • Detection of corruption. • Seeking landmarks for direct random access. • Streaming capability (i.e., the information required to parse and decode a bitstream part is available at the time at which the bitstream part is reached and does not come, e.g., at the end of the stream). • Small overhead. • Simple interleaving format with a track paradigm. We selected Xiph.Org’s [36] Ogg encapsulation format version 0 [20] as the encapsulation format for Annodex bitstreams as it meets all the requirements, has proven reliable and stable, and is an open IETF (Internet Engineering Task Force, ) standard [20]. Hier- archical formats like MPEG-4 or QuickTime were deemed less suitable as they are hierarchical file formats and therefore could not easily provide for streamable, time-accurate interleaving of multiple media and annotation tracks. To author Annodexed media, we must distinguish between files and live streams. The advantage of the former is that a file can be uploaded from the computer’s file system and annotated in a conventional authoring application. In contrast, the markup of a live Internet stream by its very nature has to be done on the fly. Annodex media files may be created in a traditional authoring application (e.g., iMovie or Adobe Premiere may easily support Annodex in the future) or through the use of CMML transcoded from metainformation collected in databases. The authoring application should support the creation of: • Structured and unstructured annotations, • Keyframe references, • Anchor points, and • URI links for media clips. Live Annodexed media streams must be created by merging clip tags with the live digital media stream. A merger application, similar to that described in Fig. 3, in- serts clip tags into the live stream at any point in time under the control of a user, e.g., by selecting a previously prepared clip tag from a list. It is expected that extending existing graphical video editing applications such as Apple’s iMovie or Adobe’s Premiere to author Annodex will be a simple task. Most already provide for specific markup to be attached to fragments of a video (sometimes also called chapters or regions ), thus extending the set of metainformation to cover keyframes, annotations, and hyperlinks should be fairly straightforward. The availability of such tools will support the uptake of Annodex for the average Web user, though computer spe- cialists can already author Annodex by using anxenc and CMML. The distribution of Annodex resources over the Internet is based on URIs, similar to the distribution of HTML pages for the World Wide Web. Annodex resources are basically accessible via any of the protocols currently used to transport media formats, e.g., RTP/RTSP [26] or HTTP [8]. The use of Annodex over RTP requires the definition of a pay- load format for Annodex, which is future work. So far we have been using Annodex only over HTTP. The basic process for the distribution and delivery of an Annodex resource is the following. A client dispatches a download or streaming request to the server with the specification of a certain URI. The server resolves the URI and starts packetizing an Annodexed media document from the requested clip or time, issuing a head tag at the start. As an alternative to streaming/downloading Annodexed media from a URI, we also envisage that different applications may prefer to retrieve only either the continuous media data or the CMML transcription. Examples are browsers that cannot handle the XML markup, and information collection applications such as search engines that do not require the media data, but just its textual representation in CMML. This is possible via the content-type flag in an HTTP client request (Fig. 4). For exploiting the rich metainformation provided by Annodex resources about media, a special media player or browser plugin is necessary. Such an application has to split an Annodex resource into its constituent header and clip tags, and the media data (the reverse of the process specified in Fig. 3). A decoder is required for the given media encoding format to display the underlying media data. While playing back the media data, the application should display the hyperlinks and the annotations for the active clip. If the displayed media data is a file and not a live stream, it is even possible to display a table of contents extracted from the annotations of the file and browse through the file based on that. The hyperlinks allow the user to freely link back and forth between Annodexed media clips, HTML pages, and other Web resources. This is transparent to the user, i.e., the user “surfs” the Annodex media resources in the way familiar from browsing the Web, because Annodex media seamlessly integrate with the existing Web. Search engines can include Annodex media resources into their search repertoire effortlessly because CMML is very similar to HTML and thus implementation of parsing support for CMML is straightforward. Indexing is performed on a per-clip basis. A search engine finds annotations in the clip tags in the desc and metatags, independent of the encoding format of the media data encapsulated in Annodex. For crawling Web resources, the search engine uses the hyperlinks given in the a element in the clip tag. Thus both indexing and crawling are supported easily with Annodex. In addition, the HTTP protocol allows one to download only the CMML markup of a published Annodexed media resource by setting in the HTTP request the Accept header with a higher priority on the media type text/x-cmml than on application/x-annodex . A conformant Annodex server will then only provide the extracted CMML content for the given Annodex resource. This prevents crawlers from creating extensive network loads. It also reduces the size of search archives, even for large amounts of published Annodex resources, because a CMML resource contains all searchable annotations for the media clips of its Annodex resource. To accumulate experience with the three developed technologies, we implemented several applications. The core applications required are a CMWeb server, a CMWeb browser, and a means to author Annodex resources. Then, a set of Annodexed audio and video content can be created and experiments on CMWeb browsing undertaken. A CMWeb search engine is also required to allow for information re- trieval to occur. We implemented all of these core applications. In this section, we report on our experiences with these applications. A CMWeb server is best implemented by extending an existing Web server with some Annodex-specific functionality. We implemented one by extending the Apache Web server [13] with a module that controls the distribution of Annodex content. It has in particular the following functionality: 1. Parses the HTTP Accept header to return the correct content type (either Annodex or CMML content). 2. Parses the URI of the HTTP request to identify a potential query component and return the requested subpart of the Annodex resource. 3. Checks if the requested Annodex resource exists on the server or has to be created on the fly from a CMML file by merging with the media stream(s) referred to in the stream tag. As we implemented most of the required authoring functionality in libraries, the Apache Annodex module mod_annodex has turned out to be a fairly small piece of code with only about 500 lines of code. It is being used on a production Apache server ( ) that serves Annodex or CMML content reliably. A CMWeb browser is a CMWeb client that can display Annodexed audio and video streams coming from a CMWeb server and provides a rich functionality for interaction with the content as enabled through the CMML tags. The use of plugins for existing Web browsers is our ultimate goal such that we can enable a full integration of Annodex content with existing Web content and retain a common browsing history. Our first prototype implementation is, however, a standalone CMWeb browser, which can be used as an external helper application to existing Web browsers. Figure 5 shows a screenshot of this prototype. The various features of this CMWeb browser include: 1. Media player : Display and transport control of the video or audio. 2. Browser : History of the browser (i.e., back and forward), reload, and stop buttons as well as URI of the resource on display. 3. Table of contents : List of the clips making up the current Annodex resource, including representative keyframes, timestamps, and short descriptions. id tags in CMML clips allow for this list of clips to be created and to be pointed to directly. The keyframes come from img tags and the text from title tags of CMML clips. 4. Annotation : Additional free-text information for the current clip stored in the desc tag in CMML clips. 5. Metadata : Additional structured text information for the current clip stored in the metatags of CMML clips. 6. Hyperlink : Attached to the current clip, a clip-dependent (i.e., time-sensitive) hyperlink points to other Web ...

Context 2

... is the format in which media with interspersed CMML markup is exchanged. Analogous to a normal Web server offering a collection of HTML pages to clients, an Annodex server offers a collection of Annodex resources. After a Web client has issued a URI request for an Annodex resource, the Web server delivers the Annodex resource, or an appropriate subpart of it according to the URI query parameters. Annodex resources conceptually consist of one or more media streams and one CMML annotation stream, interleaved in a temporally synchronized way. The annotation stream may contain several sets of clips that provide alternative markup tracks for the Annodex resource. This is implemented in CMML through a track attribute of the clip tag. The media streams may be complementary, such as an audio track with a video track, or alternative, such as two speech tracks in different languages. Figure 2 shows a conceptual representation of an example Annodex resource with three media tracks (light colored bars) and two annotation tracks (darker clips) with a header describing the complete resource (dark bar at the start). The Annodex format enables encapsulation of any type of streamable time-continuous data and is thus independent of a media compression format. It is basically a bitstream consisting of continuous media data interspersed with the structured XML markup of the CMML file. This is performed by merging the clip tags time-synchronously with the time-continuous bitstreams on authoring an Annodex bitstream. The clip tags are regarded as state changes in this respect and are valid from the time that they appear in the bitstream until another clip tag replaces them. If there is no clip that directly replaces a previous one, an empty clip tag is inserted that simply marks the end of the previous clip tag. Thus, Annodex is designed to be used as both a persistent file format and a streaming format. Figure 3 shows an example of the creation of a bitstream of an Annodexed media resource. Conceptually, the media bitstreams and the annotation bitstreams share a common timeline. When encapsulated into one binary bitstream, these data have to be flattened (serialized). CMML is designed for serialization through multiplexing. The figure shows roughly how this is performed. There are several advantages to having an integrated bitstream that includes the annotations in a time-synchronous manner with the media data. Firstly, all the information required is contained within one resource that can be distributed more easily. Also, many synchronization problems that occur with other media markup formats such as SMIL [31] are inherently solved. Also, when extracting temporal intervals from the resource for reuse, the metainformation is included in the media data, which enables one, e.g., to retain the copyright information of a clip over the whole lifetime of a reused clip. Last but not least, having a flat integrated format solves the problem of making the Annodex resource streamable. To perform the encapsulation, a specific bitstream format was required. As stated, an Annodex bitstream consists of XML markup in the annotation bitstream interleaved with the related media frames of the media bitstreams into a sin- gle bitstream. It is not possible to use straight XML as encapsulation because XML cannot enclose binary data unless encoded as Unicode, which would introduce too much overhead. Therefore, an encapsulation format that could handle binary bitstreams and textual frames was required. The following list gives a summary of the requirements for the Annodex format bitstream: • Framing for binary time-continuous data and XML. • Temporal synchronization between XML and time-continuous media bitstreams. • Temporal resynchronization after parsing error. • Detection of corruption. • Seeking landmarks for direct random access. • Streaming capability (i.e., the information required to parse and decode a bitstream part is available at the time at which the bitstream part is reached and does not come, e.g., at the end of the stream). • Small overhead. • Simple interleaving format with a track paradigm. We selected Xiph.Org’s [36] Ogg encapsulation format version 0 [20] as the encapsulation format for Annodex bitstreams as it meets all the requirements, has proven reliable and stable, and is an open IETF (Internet Engineering Task Force, ) standard [20]. Hier- archical formats like MPEG-4 or QuickTime were deemed less suitable as they are hierarchical file formats and therefore could not easily provide for streamable, time-accurate interleaving of multiple media and annotation tracks. To author Annodexed media, we must distinguish between files and live streams. The advantage of the former is that a file can be uploaded from the computer’s file system and annotated in a conventional authoring application. In contrast, the markup of a live Internet stream by its very nature has to be done on the fly. Annodex media files may be created in a traditional authoring application (e.g., iMovie or Adobe Premiere may easily support Annodex in the future) or through the use of CMML transcoded from metainformation collected in databases. The authoring application should support the creation of: • Structured and unstructured annotations, • Keyframe references, • Anchor points, and • URI links for media clips. Live Annodexed media streams must be created by merging clip tags with the live digital media stream. A merger application, similar to that described in Fig. 3, in- serts clip tags into the live stream at any point in time under the control of a user, e.g., by selecting a previously prepared clip tag from a list. It is expected that extending existing graphical video editing applications such as Apple’s iMovie or Adobe’s Premiere to author Annodex will be a simple task. Most already provide for specific markup to be attached to fragments of a video (sometimes also called chapters or regions ), thus extending the set of metainformation to cover keyframes, annotations, and hyperlinks should be fairly straightforward. The availability of such tools will support the uptake of Annodex for the average Web user, though computer spe- cialists can already author Annodex by using anxenc and CMML. The distribution of Annodex resources over the Internet is based on URIs, similar to the distribution of HTML pages for the World Wide Web. Annodex resources are basically accessible via any of the protocols currently used to transport media formats, e.g., RTP/RTSP [26] or HTTP [8]. The use of Annodex over RTP requires the definition of a pay- load format for Annodex, which is future work. So far we have been using Annodex only over HTTP. The basic process for the distribution and delivery of an Annodex resource is the following. A client dispatches a download or streaming request to the server with the specification of a certain URI. The ...

Context 3

... a temporally synchronized way. The annotation stream may contain several sets of clips that provide alternative markup tracks for the Annodex resource. This is implemented in CMML through a track attribute of the clip tag. The media streams may be complementary, such as an audio track with a video track, or alternative, such as two speech tracks in different languages. Figure 2 shows a conceptual representation of an example Annodex resource with three media tracks (light colored bars) and two annotation tracks (darker clips) with a header describing the complete resource (dark bar at the start). The Annodex format enables encapsulation of any type of streamable time-continuous data and is thus independent of a media compression format. It is basically a bitstream consisting of continuous media data interspersed with the structured XML markup of the CMML file. This is performed by merging the clip tags time-synchronously with the time-continuous bitstreams on authoring an Annodex bitstream. The clip tags are regarded as state changes in this respect and are valid from the time that they appear in the bitstream until another clip tag replaces them. If there is no clip that directly replaces a previous one, an empty clip tag is inserted that simply marks the end of the previous clip tag. Thus, Annodex is designed to be used as both a persistent file format and a streaming format. Figure 3 shows an example of the creation of a bitstream of an Annodexed media resource. Conceptually, the media bitstreams and the annotation bitstreams share a common timeline. When encapsulated into one binary bitstream, these data have to be flattened (serialized). CMML is designed for serialization through multiplexing. The figure shows roughly how this is performed. There are several advantages to having an integrated bitstream that includes the annotations in a time-synchronous manner with the media data. Firstly, all the information required is contained within one resource that can be distributed more easily. Also, many synchronization problems that occur with other media markup formats such as SMIL [31] are inherently solved. Also, when extracting temporal intervals from the resource for reuse, the metainformation is included in the media data, which enables one, e.g., to retain the copyright information of a clip over the whole lifetime of a reused clip. Last but not least, having a flat integrated format solves the problem of making the Annodex resource streamable. To perform the encapsulation, a specific bitstream format was required. As stated, an Annodex bitstream consists of XML markup in the annotation bitstream interleaved with the related media frames of the media bitstreams into a sin- gle bitstream. It is not possible to use straight XML as encapsulation because XML cannot enclose binary data unless encoded as Unicode, which would introduce too much overhead. Therefore, an encapsulation format that could handle binary bitstreams and textual frames was required. The following list gives a summary of the requirements for the Annodex format bitstream: • Framing for binary time-continuous data and XML. • Temporal synchronization between XML and time-continuous media bitstreams. • Temporal resynchronization after parsing error. • Detection of corruption. • Seeking landmarks for direct random access. • Streaming capability (i.e., the information required to parse and decode a bitstream part is available at the time at which the bitstream part is reached and does not come, e.g., at the end of the stream). • Small overhead. • Simple interleaving format with a track paradigm. We selected Xiph.Org’s [36] Ogg encapsulation format version 0 [20] as the encapsulation format for Annodex bitstreams as it meets all the requirements, has proven reliable and stable, and is an open IETF (Internet Engineering Task Force, ) standard [20]. Hier- archical formats like MPEG-4 or QuickTime were deemed less suitable as they are hierarchical file formats and therefore could not easily provide for streamable, time-accurate interleaving of multiple media and annotation tracks. To author Annodexed media, we must distinguish between files and live streams. The advantage of the former is that a file can be uploaded from the computer’s file system and annotated in a conventional authoring application. In contrast, the markup of a live Internet stream by its very nature has to be done on the fly. Annodex media files may be created in a traditional authoring application (e.g., iMovie or Adobe Premiere may easily support Annodex in the future) or through the use of CMML transcoded from metainformation collected in databases. The authoring application should support the creation of: • Structured and unstructured annotations, • Keyframe references, • Anchor points, and • URI links for media clips. Live Annodexed media streams must be created by merging clip tags with the live digital media stream. A merger application, similar to that described in Fig. 3, in- serts clip tags into the live stream at any point in time under the control of a user, e.g., by selecting a previously prepared clip tag from a list. It is expected that extending existing graphical video editing applications such as Apple’s iMovie or Adobe’s Premiere to author Annodex will be a simple task. Most already provide for specific markup to be attached to fragments of a video (sometimes also called chapters or regions ), thus extending the set of metainformation to cover keyframes, annotations, and hyperlinks should be fairly straightforward. The availability of such tools will support the uptake of Annodex for the average Web user, though computer spe- cialists can already author Annodex by using anxenc and CMML. The distribution of Annodex resources over the Internet is based on URIs, similar to the distribution of HTML pages for the World Wide Web. Annodex resources are basically accessible via any of the protocols currently used to transport media formats, e.g., RTP/RTSP [26] or HTTP [8]. The use of Annodex over RTP requires the definition of a pay- load format for Annodex, which is future work. So far we have been using Annodex only over HTTP. The basic process for the distribution and delivery of an Annodex resource is the following. A client dispatches a download or streaming request to the server with the specification of a certain URI. The server resolves the URI and starts packetizing an Annodexed media document from the requested clip or time, issuing a head tag at the start. As an alternative to streaming/downloading Annodexed media from a URI, we also envisage that different applications may prefer to retrieve only either the continuous media data or the CMML transcription. Examples are browsers that cannot handle the XML markup, and information collection applications such as search engines that do not require the media data, but just its textual representation in CMML. This is possible via the content-type flag in an HTTP client request (Fig. 4). For exploiting the rich metainformation provided by Annodex resources about media, a special media player or browser plugin is necessary. Such an application has to split an Annodex resource into its constituent header and clip tags, and the media data (the reverse of the process specified in Fig. 3). A decoder is required for the given media encoding format to display the underlying media data. While playing back the media data, the application should display the hyperlinks and the annotations for the active clip. If the displayed media data is a file and not a live stream, it is even possible to display a table of contents extracted from the annotations of the file and browse through the file based on that. The hyperlinks allow the user to freely link back and forth between Annodexed media clips, HTML pages, and other Web resources. This is transparent to the user, i.e., the user “surfs” the Annodex media resources in the way familiar from browsing the Web, because Annodex media seamlessly integrate with the existing Web. Search engines can include Annodex media resources into their search repertoire effortlessly because CMML is very similar to HTML and thus implementation of parsing support for CMML is straightforward. Indexing is performed on a per-clip basis. A search engine finds annotations in the clip tags in the desc and metatags, independent of the encoding format of the media data encapsulated in Annodex. For crawling Web resources, the search engine uses the hyperlinks given in the a element in the clip tag. Thus both indexing and crawling are supported easily with Annodex. In addition, the HTTP protocol allows one to download only the CMML markup of a published Annodexed media resource by setting in the HTTP request the Accept header with a higher priority on the media type text/x-cmml than on application/x-annodex . A conformant Annodex server will then only provide the extracted CMML content for the given Annodex resource. This prevents crawlers from creating extensive network loads. It also reduces the size of search archives, even for large amounts of published Annodex resources, because a CMML resource contains all searchable annotations for the media clips of its Annodex resource. To accumulate experience with the three developed technologies, we implemented several applications. The core applications required are a CMWeb server, a CMWeb browser, and a means to author Annodex resources. Then, a set of Annodexed audio and video content can be created and experiments on CMWeb browsing undertaken. A CMWeb search engine is also required to allow for information re- trieval to occur. We implemented all of these core applications. In this section, we report on our experiences with these applications. A CMWeb server is best implemented by extending an existing Web server with some Annodex-specific functionality. We implemented one by extending the Apache Web server [13] with a module that controls the ...