Conference PaperPDF Available

Multi-device Linear Composition on the Web Enabling Multi-device Linear Media with HTMLTimingObject and Shared Motion

Authors:

Abstract and Figures

Composition is a hallmark of the Web, yet it does not fully extend to linear media. This paper defines linear composition as the ability to form linear media by coordinated playback of independent linear components. We argue that native Web support for linear composition is a key enabler for Web-based multi-device linear media, and that precise multi-device timing is the main technical challenge. This paper proposes the introduction of an HTMLTimingObject as basis for linear composition in the single-device scenario. Linear composition in the multi-device scenario is ensured as HTMLTimingObjects may integrate with Shared Motion, a generic timing mechanism for the Web. By connecting HTMLMediaElements and HTMLTrackElements with a multi-device timing mechanism, a powerful programming model for multi-device linear media is unlocked.
Content may be subject to copyright.
Multi-device Linear Composition on the Web
Enabling Multi-device Linear Media with HTMLTimingObject and Shared Motion
Ingar M. Arntzen
Njål T. Borch
Northern Research Institute
Tromsø, Norway
ingar@norut.no
njaal@norut.no
François Daoust
Dominique Hazael-Massieux
World Wide Web Consortium
Paris, France
fd@w3.org
dom@w3.org
Copyright 2015 held by authors. Publication rights licenced to ACM. Permission to
make digital or hard copies of this work is granted given that copies bear this notice
and the full citation on the first page.
Abstract
Composition is a hallmark of the Web, yet it does not fully
extend to linear media. This paper defines linear compo-
sition as the ability to form linear media by coordinated
playback of independent linear components. We argue that
native Web support for linear composition is a key enabler
for Web-based multi-device linear media, and that precise
multi-device timing is the main technical challenge. This
paper proposes the introduction of an HTMLTimingObject
as basis for linear composition in the single-device sce-
nario. Linear composition in the multi-device scenario is
ensured as HTMLTimingObjects may integrate with Shared
Motion, a generic timing mechanism for the Web. By con-
necting HTMLMediaElements and HTMLTrackElements
with a multi-device timing mechanism, a powerful program-
ming model for multi-device linear media is unlocked.
Author Keywords
linear composition; temporal composition; interoperability;
multi-device; distributed; timing; synchronization; media
control; Web; HTMLTimingObject, HTMLMediaElement,
HTMLTrackElement.
ACM Classification Keywords
H.5.1 [Multimedia Information Systems]: Linear Media;
H.5.4 [Hypertext/Hypermedia]: Web; D.1.3 [Distributed Pro-
gramming]: Timing, control and synchronization; D.2.12
[Distributed Objects]: Interoperabilty linear components
Introduction
MediaScape: This research
is financed in part by Media-
Scape (EU, FP7-ICT-2013-
10, grant agreement 610404)
http://mediascapeproject.eu/.
The ideas presented in this
paper has impacted the
architectural approach to
multi-device Web applica-
tions currently explored by
MediaScape.
Shared Motion: a novel,
service-based approach to
multi-device timing, media
control and synchroniza-
tion. Motion Corporation
http://motioncorporation.com/
provides an online service
InMotion offering multi-
device timing for Web clients.
Client-side libraries are open
source, and the InMotion
service is publicly available.
Shared Motion is explored as
basis for multi-device linear
media in MediaScape.
W3C Multi-device Tim-
ing Community Group
https://www.w3.org/community/
webtiming/ aims to extend the
Web with native support for
multi-device timing, adopting
the Shared Motion approach
as a starting point.
Composition as a design principle is a hallmark of the Web,
ensuring reusability, extensibility, flexibility and mashup-
ability. As the Web has become a rich platform for linear
media, one would expect these benefits of composition to
apply equally well for production of linear content. In short,
linear composition implies that complex linear presenta-
tions may be constructed by combining simpler linear com-
ponents. For instance, a single Web presentation could
be formed by the coordinated playback of video [1, 7, 12],
timed meta-data run by popcornjs [9], a timed Twitter wid-
get, a map with timed geolocations, a SMIL [10] presenta-
tion, some animations based on Flash [1] or WebAnima-
tion [14], a Prezi [4] slide deck and a time-sensitive ad ban-
ner within an iFrame. Linear composition would also enable
individual components to be loaded or removed dynami-
cally during playback, either as a feature of the storyline,
or as a reaction to user input or changes in execution en-
vironment. Imagine for instance a TV program delivered to
a smart phone, adapting dynamically to the loss of WiFi by
replacing the HD video with light-weight infographics based
on timed HTML, while keeping the audio.
Unfortunately, the current support for linear composition is
weak. While the Web has wide support for linear media,
including both native and external frameworks, aspects of
timing and control are largely internal and custom to each
framework. For the programmer this means that coordi-
nating all the components and overcoming heterogeneity
may quickly become a challenge. Loading and removing lin-
ear components dynamically during playback adds further
complexity. Under such circumstances, non-functional re-
quirements such as precise and reliable synchronization will
require hard work, if at all possible.
Equally important, linear composition extends naturally to
multi-device scenarios. For example, companion device
scenarios imply coordinated playback of linear components
across TV’s, laptops, tablets and smart phones. Similarly,
collaborative viewing requires coordination of the same
linear components across multiple devices. Furthermore,
seamless workflows and presentations across multiple de-
vices may require loading and removing of linear compo-
nents dynamically, during presentation, for instance as de-
vices join and leave. This is multi-device linear composition.
In this paper we address linear composition both in single
and multi-device scenarios, and outline how the current pro-
gramming model of the Web may be transformed to support
linear composition. We argue that the key enabler for linear
composition on the Web is precise distributed timing as a
basis for coordinated playback and control. A solution for
this is already available. Shared Motion (i.e., Media State
Vector (MSV) [2]) implements distributed multi-device timing
and media control for the Web, and provides an excellent
basis for linear composition in multi-device media. We are
proposing Web support for Shared Motion through a novel
HTMLTimingObject. In this paper we particularly discuss
integration of the HTMLTimingObject with HTMLMediaEle-
ments and HTMLTrackElements, a step towards native sup-
port for linear composition on the Web.
Related Work
Linear (temporal) composition has been explored before
both in single-device and multi-device scenarios. In the
Web context, both SMIL [10] and Flash [1] are frameworks
that do linear composition internally. However, this paper
targets linear composition in a broader context; between
heterogeneous frameworks and a variety of timing-sensitive
Web components. The key issue is how coordination of lin-
ear components is performed.
A first approach to coordination is to allow general applica-
tion data to be communicated between components. For
instance, a master component may push objects to other
components, which then implement appropriate reactions.
For instance, SMIL State [5] allows communication through
shared variables. Similarly, in the multi-device scenario, me-
dia content may be pushed from a TV/STB to companion
devices, for instance using the Intranet as communication
channel. In such data-driven approaches, interfaces be-
tween components tend to be application specific. This may
imply complex integration and low flexibility. This is exem-
plified by [6] where seamless, dynamic video composition
requires considerable integration work. Still, this is only a
very basic use case for linear composition.
Single-device Regatta
Figure 1 may illustrate how a
regatta could be presented
from a collection of timed
media, e.g. official video
feed, timing info, GPS co-
ordinates and commentary,
as well as crowd-sourced
video-clips, images and Twit-
ter messages. The entire
event could then be replayed
and navigated using the me-
dia controller as common
timeline. Video-clips would
be aligned using media ele-
ments while track elements
would control map displays,
images and text.
A common improvement is to limit communication between
components by exchanging only timing and control mes-
sages, not general application data. For example, only the
currentTime property of the HTMLMediaElement [12] is
used by HTMLTrackElement [13] (or frameworks like pop-
cornjs [9]) in order to co-present timed data with media
playback. In the multi-device scenario, video playback on
Chromecast may be remote controlled by exchanging small
control messages over the Intranet, while the video data is
fetched directly from Internet. As interfaces for timing and
control can be generalized more efficiently (than application
specific data interfaces), this may improve reusability. In ad-
dition, this approach is more flexible as components may
fetch application data independently.
However, the above approaches have a weakness in com-
mon; They both introduce dependencies between compo-
nents. In short, if a component depends on another com-
ponent, for either data or timing/control, linear composition
quickly gets complicated. This observation inspires our ap-
proach. We argue that coordination must be achieved while
Figure 1: Advanced case for Web-based linear media.
Coordinating multiple media elements (blue rectangles) and track
elements (green triangles) using a shared media controller.
maintaining strict independency between components. This
paper explains how this can be done in the Web, both for
single-device and multi-device scenarios.
Linear Composition on the Web
As a starting point for discussing linear composition on the
Web, we consider three concepts, central to the Web’s na-
tive support for linear media; HTMLMediaController, HTML-
MediaElements and HTMLTrackElements [11, 12, 13].
In figure 1 one media controller (red line), two media ele-
ments (blue rectangles) and three track elements (green tri-
angles) have been combined to form a single, coordinated
presentation. Track elements depend on media elements
as they are children in the DOM. Media controllers control
the playback of the two media elements, as well as the track
elements by implication. So, linear composition is already
an important feature of the Web.
Linear composition implies that advanced linear
media may be constructed by the coordinated
playback of independent linear components.
Or, re-formulated in Web terminology:
Linear composition implies that media elements
and track elements all behave consistently, with
reference to a single, shared media controller.
Unfortunately, there are weaknesses in the current ap-
proach. In particular, the scope of the current media con-
troller is very limited, as it is designed exclusively for co-
ordination of media elements. Furthermore, the current
media controller bundles timing control with other control
aspects, such as buffering and volume control. If one me-
dia element requires buffering during playback, the media
controller automatically halts all media elements. This might
be undesired as default behavior, particularly in multi-device
scenarios. For other kinds of linear media, buffering and
volume control might not even be relevant. Another weak-
ness is precision in timing control. The media controller
depends on the timing model of media elements, essentially
a non-deterministic, pulse-based model based on repeated
firing of the timeupdate event. The lack of resolution and
predictability in this model limits precision.
We argue that generic support for linear composition re-
quires a media controller that supports high expressiveness
with respect to control primitives, and high precision with
respect to timing.
Expressiveness. A generic approach to media control
must support control primitives appropriate for various types
of linear media and a variety of use cases. For example,
slide-show presentations support next, previous or goto
whereas continuous media support play, pause and seekTo.
Animation frameworks such as SMIL [10] and WebAnima-
tions [14] even support accelerated behaviour.
Precision. A generic approach to media control must al-
low linear components to be synchronized precisely. For
Figure 2: The HTMLTimingObject is essentially an advanced
stop-watch.
instance, to support linear composition between audio and
video, lip-sync precision is required. Or, Internet radio pre-
sented by both kitchen and living room devices will produce
echo unless sub-framerate precision is supported. A deter-
ministic model for timing is required for precise synchroniza-
tion. This is especially true in a distributed scenario.
The HTMLTimingObject
A key issue for linear composition is precise and expressive
timing controls. We propose the introduction of an HTML-
TimingObject as basis for Web-based linear composition. A
draft specification for the HTMLTimingObject is being devel-
oped by the W3C Multi-device Timing Community Group at
http://webtiming.github.io/timingobject/.
The HTMLTimingObject is a very simple object, essentially
an advanced stop-watch. If started, its value changes pre-
dictably in time, until at some point later, it is paused, or
perhaps reset. The HTMLTimingObject may be queried for
its value at any time. In terms of implementation, the HTML-
TimingObject is a fairly trivial wrapping around the system
clock. This means that it shares properties of the system
clock in terms of resolution and predictability. This makes
Figure 3: Media elements (blue rectangles) and track elements
(green triangles) directed by a single HTMLTimingObject (red line).
the HTMLTimingObject a sound basis for precise linear syn-
chronization.
The HTMLTimingObject is more expressive than a tradi-
tional stop-watch. It supports any velocity or acceleration,
and may jump to any position on the timeline. In fact, the
HTMLTimingObject simply implements linear motion along
a unidimensional axis. An elegant implementation is pro-
vided by the concept of Media State Vectors (MSV) [2]. At
any point in time, position, velocity or acceleration may be
requested to change. Querying the HTMLTimingObject re-
veals not only its current value (position) but also its velocity
and acceleration at that moment. This detailed information
is again helpful in precise synchronization, and the expres-
siveness of the underlying mathematical model implies that
a wide variety of control primitives may be supported.
We are not the first to define timing controls for linear me-
dia. Similar constructs have been explored in both academia
and industry from the 70’s and onwards. Indeed, any frame-
work for linear media would maintain similar constructs in-
ternally. Instead, the novelty lies in representing timing as
an explicit resource on the Web, independent of framework,
thereby creating a basis for interoperability as well as multi-
device timing. The latter is the subject of the next section.
Figure 4: Media elements (blue rectangles) and track elements
(green triangles) distributed or duplicated across devices, each
device with its own HTMLTimingObject (red line).
Figure 3 illustrates linear composition, with the HTML-
TimingObject (red line) replacing the HTMLMediaController
as director. The illustration also shows how HTMLMedia-
Elements and HTMLTrackElements equally interface directly
with the timing object. Each of these linear components
will monitor the timing object, and implement appropriate
reactions whenever it pauses, resumes, jumps or speeds
up. Linear components may also enforce control over the
shared timing object, by requesting it to pause, resume etc.
Such requests trigger effects on all components in unison.
Crucially, all linear components may remain mutually inde-
pendent, and even agnostic of each others existence. This
is possible as they do not communicate directly, but only in-
directly through a shared timing object. This independency
between linear components is crucial for flexible and dy-
namic linear composition.
Multi-device Linear Composition for Web
Importantly, the concept of linear composition extends nat-
urally to multi-device scenarios. Essentially, we want to go
from single-device to multi-device by scattering or duplicat-
ing linear components across devices. At the same time,
we need to go from single-device playback, to simultane-
ous, multi-device playback.
Figure 4 illustrates how two media elements (blue rectan-
gles) and two track elements (green triangles) may be split
across three devices. Note also that this multi-device sce-
nario, particularly B), demonstrates why the current depen-
dency between track elements and media elements is not
appropriate. In this illustration, track elements are promoted
as standalone programming constructs depending directly
on the HTMLTimingObject, just like media elements.
In order to support multi-device linear composition, the chal-
lenge is to ensure that HTMLTimingObjects on multiple de-
vices are kept in synchrony. In the single-device scenario,
all linear components were using a single HTMLTiming-
Object as shared media control. We propose to extend this
notion to the multi-device scenario.
In multi-device, linear media, distributed me-
dia elements and track elements are equally
connected to a single, shared timing object.
Furthermore, since we are designing for the Web, shared
timing objects should be available wherever and when-
ever the Web is available. It follows that technical solutions
based on services or features of local networks, specific
network carriers, NAT traversal etc., are not appropriate.
Also, in line with the client-server architecture of the Web,
we prefer a centralized, service- based solution. So, we
propose the concept of online timing objects, hosted by web
services and available for all connected devices.
Multi-device Regatta
Figure 4 may illustrate how a
regatta could be presented
in a multi-device scenario.
Interactive race infographics
and the regatta map may be
hosted by an iPad, while a
smart TV presents the main
video feed. A smart phone
may present time-aligned
video-clips, images and com-
ments, while at the same
time serving as an input-
device for user-generated
content. Finally, media con-
trol is available from all
devices. For instance, a
touch-sensitive regatta time-
line on the iPad may support
easy timeshifting, as would
a simple progress bar on the
smart phone. Media control
affects all components in
unison, thereby providing
consistent linear composition
across multiple devices.
Figure 5 illustrates a single, online timing object (red line),
shared between distributed media elements (blue rectan-
gles) and track elements (green triangles). The HTMLTiming-
Objects on each device (red lines) serve as local represen-
tation for the shared, online timing object. As the HTML-
TimingObject encapsulates synchronization with online
Figure 5: HTMLTimingObjects (red lines) mediating access to an
online timing object (red line). HTMLTimingObjects on different
devices connect and synchronize individually.
timing objects, media elements and track elements may
readily support linear composition in multi-device as well as
single-device media. In principle, distributed synchroniza-
tion would only require the programmer to specify a valid
URL for the source attribute of the HTMLTimingObject.
Shared Motion
A solution is already available for the implementation of
synchronization with online timing objects. Shared Motion
is a generic mechanism for distributed timing on the Web.
Similar to the HTMLTimingObject, it is based on Media
State Vectors (MSV) [2], a representation of determinis-
tic, unidimensional motion. The synchronization protocol of
Shared Motion is based on ad-hoc, application-level clock
synchronization, and allows Web clients to maintain sub-
framerate precision [2, 3] across the Internet. Shared Mo-
tion is also extremely scalable, allowing a vast number of
clients to participate in multi-device linear media presenta-
tions. Shared Motion does not depend on NTP [8] synchro-
nization, and works independent of network carrier, OS or
browser type (without requiring plugins). Though Shared
Motion is designed for the Web it is not restricted to the
Web. As such, Shared Motion is an enabling mechanism for
distributed linear composition, applicable to all connected
clients, from web-browsers on laptop computers to native
applications on embedded devices.
Evaluation
In order to evaluate HTMLTimingObjects and Shared Mo-
tion as basis for linear composition, we have focused on
integration with HTMLMediaElement and HTMLTrackEle-
ments as a first step. A JavaScript wrapper is provided
for multi-device synchronization of HTML5 Audio/Video,
demonstrating synchronization errors < 10ms [3] across the
Internet. A JavaScript re-implementation of the HTMLTrack-
Element demonstrates distributed presentation of timed
data with millisecond precision (relative to HTMLTimingOb-
ject). The precision, reliability and simplicity of the proposed
programming concepts further demonstrates the feasibil-
ity and value of this approach. Demonstrations have been
made publicly available by Motion Corporation and the W3C
Multi-device Timing Community Group.
In this work we have also identified a series of weaknesses
with respect to HTML support for timed operation. We aim
to address these weaknesses through the W3C Multi-device
Timing Community Group, thereby improving the Web as a
platform for timed media.
The HTMLMediaElement is currently not optimized for ex-
ternal control. In particular, media elements should com-
pensate for internal delays (buffering etc.) by adjusting their
internal playback offset according to an external HTML-
TimingObject. In addition, to maintain synchrony, it is nec-
essary for media elements to compensate for playback drift
relative to the HTMLTimingObject. Media elements also
behave differently with different browsers, different media
types and on different architectures. It is hard to maintain
a library masking all these differences. Even worse, prop-
erties affecting synchronization are sensitive to changes
across browsers updates. So, ideally these problems should
be addressed by media elements internally, thereby im-
plementing native support for timed operation. The multi-
device timing group will work on a test suite for HTMLMedia-
Elements, exposing such timing issues.
The HTMLTrackElement too would benefit from improve-
ments with respect to precision. Our initial tests indicate
that cue events are emitted rather coarsely. In Chrome ver-
sion 39 for example, events appear to fire about 150-250
milliseconds too late according to the video currentTime.
Our improved version of the track element reliably delivers
event upcalls at the correct millisecond. Furthermore, the
current track element provides events only during playback,
not as a response to seekTo. Our implementation guaran-
tees enter and exit events to always be consistent with any
kind of media control supported by Shared Motion, includ-
ing acceleration. Consistency of enter and exit events ad-
ditionally extends to dynamic cue changes. For instance, a
timed subtitle may safely be added, removed, or prolonged
in time- span, during playback, without introducing added
complexity for the programmer.
If properly integrated with the HTMLTimingObject, media
elements and track elements would be readily available for
precise and flexible linear composition in the single device
scenario. Additionally, integration of SharedMotion with the
HTMLTimingObject would immediately enable support for
multi-device linear composition.
Summary
This paper explains the importance of linear composition
for Web-based multi-device media, and identifies precise,
expressive multi-device timing control as the key technical
enabler. The proposed solution involves the introduction of
a HTMLTimingObject and integration with Shared Motion
for multi-device timing. The approach is verified by integra-
tion of HTMLMediaElements and HTMLTrackElements with
SharedMotion. The precision, reliability and simplicity of the
proposed programming concepts support our claim that this
approach unlocks a highly powerful programming model
supporting multi-device linear composition on the Web.
References
[1] Adobe. 2015. Adobe Flash runtimes. https://www.adobe.
com/products/flashruntimes.html. (2015).
[2] Ingar M. Arntzen, Njål T. Borch, and Christopher P.
Needham. 2013. The Media State Vector: A Uni-
fying Concept for Multi-device Media Navigation. In
Proceedings of the 5th Workshop on Mobile Video
(MoVid ’13). ACM, New York, NY, USA, 61–66.
DOI:
http://dx.doi.org/10.1145/2457413.2457427
[3] Njål T. Borch and Ingar M. Arntzen. 2014. Dis-
tributed Synchronization of HTML5 Media. Tech-
nical Report 15. Northern Research Institute. http:
//norut.no/nb/node/3041
[4] Prezi Inc. 2015. Prezi for Presentation. http://prezi.com/.
(2015).
[5] Jack Jansen and Dick C.A. Bulterman. 2008. Enabling
Adaptive Time-based Web Applications with SMIL
State. In Proceedings of the Eighth ACM Symposium
on Document Engineering (DocEng ’08). ACM, New
York, NY, USA, 18–27.
DOI:
http://dx.doi.org/10.1145/
1410140.1410146
[6] Jack Jansen, Pablo Cesar, Rodrigo Laiola Guimaraes,
and Dick C.A. Bulterman. 2012. Just-in-time Per-
sonalized Video Presentations. In Proceedings of the
2012 ACM Symposium on Document Engineering (Do-
cEng ’12). ACM, New York, NY, USA, 59–68.
DOI:
http://dx.doi.org/10.1145/2361354.2361368
[7] Microsoft. 2015. Microsoft Silverlight. http://www.
microsoft.com/silverlight/. (2015).
[8] David. L. Mills. 1991. Internet time synchronization:
the network time protocol. IEEE Transactions on Com-
munications 39, 10 (Oct 1991), 1482–1493.
DOI:
http://dx.doi.org/10.1109/26.103043
[9] Mozilla. 2015. Popcorn.js The HTML5 Media Frame-
work. http://popcornjs.org/. (2015).
[10] Patrick. Schmitz. 2002. Multimedia meets computer
graphics in SMIL2.0: a time model for the web. In Pro-
ceedings of the 11th international conference on World
Wide Web (WWW ’02). ACM, New York, NY, USA, 45–
53.
DOI:
http://dx.doi.org/10.1145/511446.511453
[11] W3C. 2015a. HTML5 Media Controller. http:
//dev.w3.org/html5/spec-preview/Overview.html#
synchronising-multiple-media-elements. (2015).
[12] W3C. 2015b. HTML5 Media Elements. http://dev.w3.org/
html5/spec-preview/media-elements.html. (2015).
[13] W3C. 2015c. HTML5 Track Element. http://dev.w3.org/
html5/spec-preview/the-track-element.html. (2015).
[14] W3C. 2015d. Web Animations. http://www.w3.org/TR/
web-animations/. (2015).
... UI-independent sequencing simplifies integration of new data types into visual and interactive components. Integration with an external timing object [7] ensures that media components based on the Sequencer may trivially be synchronized and remote controlled, both in single-page media presentations as well as global, multi-device media applications [5,6,7,16]. A JavaScript implementation for the Sequencer is provided based on setTimeout, ensuring precise timing and reduced energy consumption. ...
... Instead, we advocate a programming model where timing and sequencing functionality are made available as independent, generic programming tools, applicable across application domains. The timing object [5,7] is the fundamental building block of this programming model, defining a unifying foundation for timing, synchronization and control. This paper presents the Sequencer, a generic sequencing tool as an additional building block for timed media applications. ...
... Support for dynamic sequencing allows dynamic data sources to be used without introducing any additional complexity for the programmer. Finally, the timing object supports distributed timing through Shared Motion [5,6,16]. In short, this opens up multi-device sequencing to any connected Web agent, independent of data format, delivery mechanism or UI framework. ...
Conference Paper
Media players and frameworks all depend on the ability to produce correctly timed audiovisual effects. More formally, sequencing is the process of translating timed data into correctly timed presentation. Though sequencing logic is a central part of all multimedia applications, it tends to be tightly integrated with specific media formats, authoring models, timing/control primitives and/or predefined UI elements. In this paper, we present the Sequencer, a generic sequencing tool cleanly separated from data, timing/control and UI. Data-independent sequencing implies broad utility as well as simple integration of different data types and delivery methods in multimedia applications. UI-independent sequencing simplifies integration of new data types into visual and interactive components. Integration with an external timing object [7] ensures that media components based on the Sequencer may trivially be synchronized and remote controlled, both in single-page media presentations as well as global, multi-device media applications [5, 6, 7, 16]. A JavaScript implementation for the Sequencer is provided based on setTimeout, ensuring precise timing and reduced energy consumption. The implementation is open sourced as part of timingsrc [2, 3], a new programming model for precisely timed Web applications. The timing object and the Sequencer are proposed for standardization by the W3C Multi-device Timing Community Group [20].
... Temporal interoperability implies that multiple, possibly heterogeneous media com- ponents may easily be combined into a single, consistently timed media experi- ence [5]. We argue that temporal interoperability must be promoted as a principal feature of the Web, and finding the right approach to media synchronization is key to achieving this. ...
Chapter
The Web is a natural platform for multimedia, with universal reach, powerful backend services, and a rich selection of components for capture, interactivity, and presentation. In addition, with a strong commitment to modularity, composition, and interoperability, the Web should allow advanced media experiences to be constructed by harnessing the combined power of simpler components. Unfortunately, with timed media this may be complicated, as media components require synchronization to provide a consistent experience. This is particularly the case for distributed media experiences. In this chapter we focus on temporal interoperability on the Web, how to allow heterogeneous media components to operate consistently together, synchronized to a common timeline and subject to shared media control. A programming model based on external timing is presented, enabling modularity, interoperability, and precise timing among media components, in single-device as well as multi-device media experiences. The model has been proposed within the W3C Multi-device Timing Community Group as a new standard, and this could establish temporal interoperability as one of the foundations of the Web platform.
... In recent years a growing body of research has analysed and verified the benefits that so-called Time Awareness can bring to a broad range of application domains [4], [12], [11], [2]. Of particular interest is the TAACCS Interest GRoup (Time Aware Applications, Computers and Comm Systems, that has published a NIST white paper [27]. ...
Conference Paper
Time synchronisation plays a critical role in time-sensitive distributed applications. While a variety of such applications exist across many domains, one particular set of applications where improved time synchronisation can lead to significant benefits, particularly with respect to QoE (Quality of Experience), is multimedia applications. While time synchronisation is not a new challenge, advances in wireless technologies have drastically transformed network infrastructures. 802.11 wireless networks increasingly represent the last hop within the ever expanding Internet and whilst users expect the same levels of multimedia QoE as exist over wired networks, the reality of moving back to contention based access leaves many disappointed. This transformation of networks has also proven problematic for time synchronisation protocols that were designed for wired infrastructures. Wireless networks, particularly contention based networks, can be the source of very significant non-deterministic packet latencies. In certain scenarios, such latencies can greatly degrade the performance of time synchronisation. This work details and validates a technique that can be used to determine the latency of time messages in real-time as they traverse an 802.11 wireless link. Knowledge of these latencies can be used to greatly reduce the error in a dataset employed by time synchronisation protocols such as NTP and, thus, improve their performance. Experimental results confirm error reductions of up to 90% in a dataset and prove that the use of this technique can deliver time accuracies akin to those achievable over wired networks. This in turn can greatly benefit users by enabling multimedia applications to benefit from the continued use of time synchronisation for QoE management. We outline two such scenarios, one where time synchronisation is used to prioritise VoIP traffic within an Access Point and a second where the aim is to use time synchronisation to optimise jitter buffer strategies for WebRTC.
Conference Paper
In this work, we present a case study in which participants used a mobile application for collaborative record presentations. We developed an application for Android-enabled devices to investigate the usability and design requirements of a mobile and collaborative capture system. Our main concern is to facilitate collaboration and create an enhanced result without adding complexity in relation to the individual capture task. In this way, we focused on problems related to the usability and to the awareness information that enables users to conduct an opportunistic recording. We report our case study results and discuss design requirements we identified for the collaborative recording of presentations by users in possession of smartphones and tablets.
Technical Report
Full-text available
In this report we analyze the quality of synchronization we can expect when synchronizing HTML5 audio and video on multiple devices using Shared Motion. We demonstrate that the concept of Shared Motion enables sub-frame synchronization for video, and near perfect synchronization for audio. Experiments are conducted in real world scenarios.
Conference Paper
Full-text available
Using high-quality video cameras on mobile devices, it is relatively easy to capture a significant volume of video content for community events such as local concerts or sporting events. A more difficult problem is selecting and sequencing individual media fragments that meet the personal interests of a viewer of such content. In this paper, we consider an infrastructure that supports the just-in-time delivery of personalized content. Based on user profiles and interests, tailored video mash-ups can be created at view-time and then further tailored to user interests via simple end-user interaction. Unlike other mash-up research, our system focuses on client-side compilation based on personal (rather than aggregate) interests. This paper concentrates on a discussion of language and infrastructure issues required to support just-in-time video composition and delivery. Using a high school concert as an example, we provide a set of requirements for dynamic content delivery. We then provide an architecture and infrastructure that meets these requirements. We conclude with a technical and user analysis of the just-in-time personalized video approach.
Conference Paper
Full-text available
In this paper we examine adaptive time-based web applications (or presentations). These are interactive presentations where time dictates the major structure, and that require interactivity and other dynamic adaptation. We investigate the current technologies available to create such presentations and their shortcomings, and suggest a mechanism for addressing these shortcomings. This mechanism, SMIL State, can be used to add user-defined state to declarative time-based languages such as SMIL or SVG animation, thereby enabling the author to create control flows that are difficult to realize within the temporal containment model of the host languages. In addition, SMIL State can be used as a bridging mechanism between languages, enabling easy integration of external components into the web application. Categories and Subject Descriptors
Conference Paper
Full-text available
Multimedia scheduling models provide a rich variety of tools for managing the synchronization of media like video and audio, but generally have an inflexible model for time itself. In contrast, modern animation models in the computer graphics community generally lack tools for synchronization and structural time, but allow for a flexible concept of time, including variable pacing, acceleration and deceleration and other tools useful for controlling and adapting animation behaviors. Multimedia authors have been forced to choose one set of features over the others, limiting the range of presentations they can create. Some programming models addressed some of these problems, but provided no declarative means for authors and authoring tools to leverage the functionality. This paper describes a new model incorporated into SMIL 2.0 that combines the strengths of scheduling models with the flexible time manipulations of animation models. The implications of this integration are discussed with respect to scheduling and structured time, drawing upon experience with SMIL 2.0 timing and synchronization, and the integration with XHTML.
Conference Paper
This paper presents the concept of the Media State Vector (MSV), an implementation of uni-dimensional motion in real time. The MSV is intended as a general representation of media navigation and a basis for synchronization of multi-device media presentations. The MSV is motivated by the idea that media navigation can be decoupled from media content and visual presentation, and shared across a network. Implementation of the MSV concept for the Web allows us to construct navigable, synchronized, multi-device, multimedia presentations, spanning computers across the Internet. In particular, media presentations may be hosted by regular Web browsers on a range of devices, including smart phones, pads, laptops and smart TVs. Our proof of concept implementation bases its synchronization accuracy on primitive, centralized, ad-hoc, application-level clock synchronization. Still, inter-client synchronization error of about 33 ms is demonstrated between three screens in London (UK), synchronized via a server in Tromsø (Norway).
Article
The network time protocol (NTP), which is designed to distribute time information in a large, diverse system, is described. It uses a symmetric architecture in which a distributed subnet of time servers operating in a self-organizing, hierarchical configuration synchronizes local clocks within the subnet and to national time standards via wire, radio, or calibrated atomic clock. The servers can also redistribute time information within a network via local routing algorithms and time daemons. The NTP synchronization system, which has been in regular operation in the Internet for the last several years, is described, along with performance data which show that timekeeping accuracy throughout most portions of the Internet can be ordinarily maintained to within a few milliseconds, even in cases of failure or disruption of clocks, time servers, or networks
Microsoft Silverlight
  • Microsoft
Microsoft. 2015. Microsoft Silverlight. http://www. microsoft.com/silverlight/. (2015).