An event-based data distribution mechanism for collaborative mobile augmented reality and virtual environments

Abstract

The full power of mobile augmented and virtual reality systems is realized when these systems are connected to one another to immersive virtual environments, and to remote information servers. Connections are usually made through wireless networks. However, wireless networks cannot guarantee connectivity and their bandwidth can be highly constrained. The authors present a robust event-based data distribution mechanism for mobile augmented reality and virtual environments. It is based on replicated databases, pluggable networking protocols, and communication channels. We demonstrate the mechanism in the Battlefield Augmented Reality System (BARS) situation awareness system, which is composed of several mobile augmented reality systems, immersive and desktop-based virtual reality systems, a 2D map-based multi-modal system, handheld PCs, and other sources of information.

Full text

Page 1

An Event-Based Data Distribution Mechanism for Collaborative Mobile
Augmented Reality and Virtual Environments
Dennis Brown, Simon Julier, Yohan Baillot
ITT Advanced Engineering & Sciences
2560 Huntington Ave
Alexandria, VA 22303
fdbrown,julier,baillotg@ait.nrl.navy.mil
Mark A. Livingston
Naval Research Laboratory
4555 Overlook Ave SW
Washington, DC 20375
markl@ait.nrl.navy.mil
Abstract
The full power of mobile augmented and virtual real-
ity systems is realized when these systems are connected to
one another, to immersive virtual environments, and to re-
mote information servers. Connections are usually made
through wireless networks.
cannot guarantee connectivity and their bandwidth can be
highly constrained. In this paper we present a robust event-
based data distribution mechanism for mobile augmented
reality and virtual environments. It is based on replicated
databases, pluggable networking protocols, and communi-
cation channels. We demonstrate the mechanism in the Bat-
tlefield Augmented Reality System (BARS) situation aware-
ness system, which is composed of several mobile aug-
mented reality systems, immersive and desktop-based vir-
tual reality systems, a 2D map-based multi-modal system,
handheld PCs, and other sources of information.
However, wireless networks
1. Introduction
As computers have become smaller and more powerful,
the concept of portable, high performance system computer
system for augmented and virtual reality has become feasi-
ble. State-of-the-art (2002) laptops boast processor, mem-
ory, disk, and graphics hardware that rival supercomputers
from only five years ago. The full power of these systems
is realized when these systems are networked with one an-
other and with remote information servers. For example, a
user walking down a street might be able to see the loca-
tions of other users, up-to-date information about prices in
shop windows, and even email [4]. In our research on the
Battlefield Augmented Reality System (BARS) [9] [11], we
are focused on the problem of developing information sys-
tems able to provide users with “situation awareness” —
data about their environment and its contents.
The problem of distributing data to such users is some-
what unusual. Most research into distributed virtual or aug-
mented reality focuses on the support of common, consis-
tent virtual worlds, typically maintained by a small number
of users. However, the objective of BARS is to support a
consistent information space. Therefore, objects tend to be
less complicated and updates occur less frequently than in
virtual environments. Furthermore, it requires a unified ar-
chitecture that allows transport of general state information
that can be used for many purposes, such as the obvious task
of distributing the virtual object database, as well as more
general uses such as “remote control” of applications. This
system must also handle the poor network connectivity that
cansometimesbeencounteredinmilitaryoperations. Given
these parameters, we have developed a robust, flexible, and
general event-based networking infrastructure for data dis-
tribution. The mechanism builds upon three techniques:
distributed databases, pluggable transport protocols, and a
high-level management technique known as channels.
The structure of this paper is as follows. The problem
statement and a survey of existing literature is given in Sec-
tion 2. Section 3 describes the event distribution system.
Conclusions and future work are discussed in Section 4.
2. Problem Statement
The Battlefield Augmented Reality System (BARS) is a
collaborative mobile augmented reality system designed to
improve the situation awareness and the coordination be-
tween a team of mobile users. By situation awareness, we
mean that each user obtains a better understanding of the
environment. The types of data include the names of build-
ings, routes, objectives, and the locations of other users.
While short-range radio communications can accomplish
much of this, the passive and natural display paradigm of
augmented reality (AR) makes distribution of this type of
information important for mobile AR systems.

Page 2

The hardware of one of our prototype systems is shown
in Figure 1. It consists of a mobile computer, either an em-
bedded PC with a high-end graphics card or a laptop with
high-end graphics built in. A see-through SVGA display
is worn by the user; we have built systems with the Sony
GlasstronTMand the Microvision NomadTMretinal display.
The system supports spatialized audio through software, us-
ing commodity sound cards and headphones. The user op-
erates the system using a cordless mouse and a wrist key-
board. A Global Positioning System (GPS) receiver han-
dles position tracking while an inertial device handles ori-
entation tracking. A camera can be used for tracking or for
sending video reports to a base station. Wireless 802.11b
networking is used for data distribution and GPS correc-
tions1. The BARS mobile user sees graphics on or next to
the real objects they are intended to augment, as well as
status information such as a compass and messages from
other users. Figure 2 shows an output from the system —
the parking lot is augmented to highlight the neighboring
building and a fellow BARS user.
Figure 1. The BARS Wearable
This application introduces number of characteristics
that impact the distribution of information and events be-
tween users:
? The objective is to provide information, not a consis-
tent virtual world. The environment can be viewed
to be populated by a set of objects which are self-
contained entities and other types of discrete data.
Each object can be relatively simple. It is not neces-
sary to transmit complicated geometric objects or be-
haviors (such as animation). The latency in the update
of an object or an entity is a secondary consideration.
1When some future version of BARS is used in real operations, com-
munication would likely happen over the US military’s communications
system of that time, its construction probably influenced by the current
research described by North et.al. [13]. However, it is likely that any de-
ployed system will still have problems with connectivity and bandwidth in
built-up areas.
Figure 2. A Sample Augmentation
? Data distribution between users can be heterogeneous.
Different users might carry out different tasks and thus
have different information requirements.
? The distribution system should facilitate collaboration
between different users. In addition to environmental
data, the distribution system must support the propaga-
tion of meta-data such as task assignments, objectives,
and messages.
? Users should have the ability to create reports and up-
date entities in the database. For example, a user might
observe that an environmental feature (such as a vehi-
cle) is not where the database indicates it should be.
The user should have the ability to move the object to
its correct location.
? Network connectivity is unreliable. As a user moves
from location to location, reception strength will vary
and the bandwidth can be limited and can be time
varying (depending on signal strength and number of
users).
Given its potential importance, a great deal of research
has been carried out into distributed systems for virtual and
augmented reality.
The Naval Postgraduate School has been working on
large-scale distributed virtual environments for over a
decade, their first version of NPSNET being shown in
1991. The data distribution in the current version, NPSNET
V [3], uses replicated data and the model-view-controller
paradigm to maintain it. Users of the system see a view of
the entities (models) drawn by a rendering system. These
entities are modified by the protocols (controllers) that
translate network messages into state changes. New pro-
tocols can be loaded at run-time to extend the behaviors of
entities. The packets are sent via multicast, and the system
allows many protocols to share a single multicast socket.
Network traffic is controlled with an area-of-interest man-
ager.

Page 3

The Distributed Interactive Virtual Environment (DIVE)
system [5] is designed to scale to many participants while
maintaining a high level of interactivity at each site. It is
based on a peer-to-peer architecture that uses reliable mul-
ticast to maintain a database that is replicated at each peer.
Using a heirarchical partitioning of the database, only part
of the database may be replicated at some peers. To main-
tain highly interactive rates, some copies of the database
may be updated faster than others, but there are mechanisms
to ensure equality over time.
MASSIVE-2 [6] uses the concept of a local object, or
artifact, andremoteviewsofthatartifact. Theseartifactsare
paged in and out of remote applications. A spatial model of
interaction is used in that a remote application will have a
focus that defines a subgroup of the artifacts to page in and
out, instead of copying the entire database. Artifact state
changes are distributed using reliable multicast.
Not all work has been solely for immersive virtual envi-
ronments. Some collaborative AR systems have been built,
typically for small groups of co-located users. One example
is Shared Space system [2] for computer-supported collabo-
rative work using AR. Typically, users are operating closely
with one another and latency requirements are extremely
high. Another system designed for augmented and virtual
reality is COTERIE [12], which provides useful semantics
through its notions of Distributed Shared Memory and Call-
back Objects. Distributed Shared Memory allows many ap-
plications to shared the same data objects, while Callback
Objects allow applications to be notified of changes to those
data objects — an event-like mechanism. The Studierstube
system [16] extends the concept through its use of an exten-
sible scenegraph of application objects that is distributed to
all users.
3. The BARS Event Distribution System
The BARS distribution system is based entirely on the
concept of events. Events are used both to instantiate ob-
jects (in effect, transmit a view of a database between sys-
tems), to send information to update preexisting objects,
and to provide other non-database status information (such
as a new objective for an individual user). The event dis-
tribution system is based on three components: event trans-
porters, replicated object repositories, and communication
channels. We will describe these components, as well as
the use of bridge applications to communicate with outside
virtual and augmented reality systems, and some other uses
of the event distribution system.
3.1. Event Transportation
The heart of the event transportation system is the Object
and Event Manager. The Object and Event Manager is re-
Figure 3. Event distribution within an applica-
tion: Arrows show event movement.
sponsible for dispatching events within an application and
distributing those events to remote applications.
When the Object and Event Manager receives an event,
it places it on an asynchronous event queue. The event dis-
patching thread delivers the event to all the listeners that are
subscribed to receive the specified event type. The event
dispatching mechanism maintains two sets of data — the
set of valid event/listener pairs, and the set of listeners reg-
istered for each event type. Because the event system has
its roots in the Java AWT event model [17] we leverage the
Reflection API to achieve these steps. The type of each
event is specified by its class. For each event type, a listener
is defined as well. When a listener is registered with the
object and event manager, its interface is queries and it is
registered to receive all event types for which its interface is
compatible. Therefore, it is possible to dynamically extend
the set of event and listener types at runtime.
The following is an example of the life of an event within
an application. A thread handling the position and orien-
tation trackers will update the user’s position by calling a
method in the user object to set its attitude based on data
gathered from the trackers. This method in turn creates an
event encapsulating the change to the attitude and sends it
to the dispatcher. The event arrives in the dispatcher’s in-
coming queue and is soon processed. The dispatcher sends
the event to all listeners, including the object itself, as well
as the graphics system (to update the viewpoint) and the fil-
ter (to update what objects are being drawn), and any other
registered listeners. Note that the object’s attitude isn’t set
until it receives the event back from the dispatcher (the al-
ternative is to set the position at the same time the event is
set) — this way, the order of events is retained by going
through the event queue in the dispatcher. Figure 3 shows
the flow of events within an application.
We have described how events propagate within a single
application instance. We extended this event mechanism
to allow many separate BARS applications to trade events
by creating Event Transporters that allow Object and Event
Managers in different application instances to send events

Page 4

to and receive events from each other using Internet Pro-
tocol (IP). If an event is tagged as distributed, an Event
Transporter serializes the event and broadcasts it to other
applications. The Event Transporters in remote applica-
tions synthesize the event object, and dispatch it on those
applications’ event queues. Our system uses several types
of transporters, based on IP multicast, the Lightweight Re-
liable Multicast Protocol (LRMP) from INRIA [10], and
a combination protocol we call the Selectively Unreliable
Multicast Protocol (SUMP) that combines IP multicast and
LRMP. Typically, application instances use SUMP on the
localnetwork. Tocommunicateoutsideofthelocalnetwork
(where multicast is typically filtered out) a TCP/IP trans-
porter and bridge are used (described later in the Bridges
section). Because of the connectionless nature of IP multi-
cast, thedistributionisrobustinthatthenetworkconnection
can come and go and the user application still functions, al-
though without network updates at some times.
As events are created, they are tagged “reliable” or “un-
reliable” designating how they should be sent. Object cre-
ation and deletion are always sent reliably. Object changes
are sent reliably or unreliably based first on whether the
change is relative or absolute. Relative changes have an
ordering and each one is important, so those are sent re-
liably. Absolute changes, such as the constant updates of
a user position, are mostly sent unreliably since if one is
missed, the next would overwrite it anyway — periodically
these changes are sent reliably. This policy makes the as-
sumption that the implementation of IP networking in a real
operation may drop IP packets often, making reliable multi-
cast expensive, and so we don’t send events reliably unless
we think it is truly necessary. Figure 4 shows the flow of
events between applications.
3.2. Replicated Object Repositories
An object repository inside a BARS application holds
the data for that application.
mostly static models of the physical surroundings (build-
ings, streets, points of interest, etc), dynamic avatars for
users and entities, and objects created as communications,
such as reports of enemy locations, routes for users to fol-
low, and digital ink. This database is replicated in whole or
in part for every BARS application.
When an application starts, it loads an initial set of ob-
jects from a number of sources, including data files, other
applications already running on the network, and files spec-
ified on the command line. This initial set of objects typi-
cally consists of street labels, landmarks, building informa-
tion, and other terrain-like information, as well as an initial
set of objectives, routes, and phase markers for the current
task. Therefore, since the user will have some form of the
database to start, and everything else in the wearable BARS
This data consists of the
system is self-contained, the user will have a working AR
system even if all network connectivity is lost during an op-
eration.
Although network limitations may hamper wireless
communications for the mobile users, there are very few
limitations on the base users. Their applications run on sta-
tionary VR systems such as a desktop computers, 3D work-
benches, and immersive VR rooms. Using the same distri-
bution system, they will have high levels of detail and inter-
action by taking advantage of the better bandwidth for repli-
cating more objects and seeing change events at a higher
frequency.
3.3. Channels
If simply implemented as described above, the event dis-
tribution mechanism will send all events to every applica-
tion, which for example would replicate the entire database
inside every application. As demonstrated by the works of
others described earlier, creating copies of every object for
every user and updating those replicas would soon swamp
the network with information many users would not care
about anyway. So, like those systems, we have devised a
way of partially replicating the database to minimize the
amount of unwanted data distributed to each application.
In creating this replication mechanism, we first looked
at the uses of BARS that should drive our policies. One
condition to consider is that a mobile user can only see so
much and deal with information in a relatively small radius,
so we looked at a spatial area-of-interest mechanism. It is
not necessarily the case that a mobile user only cares about
objects that can be seen from his or her current position in
the real world; for example, our mobile application includes
an overhead map mode in which the user can zoom out to
an arbitrary height and would want to see objects within a
possibly huge radius around the current position. However,
it seems that there would be few situations in which a mo-
bile user would care about objects very faraway, sofor most
situations, a simple area-of-interest mechanism could work.
However, another condition is the type of information
being distributed. Even if some objects are near to a mobile
user, they may have no importance and could be distract-
ing. Or, they may indeed be too far away to be seen, but
very important, such as with possible sniper locations. For
these cases, a simple area-of-interest mechanism isn’t suffi-
cient. In an earlier paper [8], we described a filtering mech-
anism for mobile augmented reality. This filtering mecha-
nism works on the object repository within an application.
It is sufficient for its original goal of not showing users ob-
jects in which they have no interest in order to reduce dis-
play clutter. However, it simply hides objects from the user
— it does not actually control whether or not the applica-
tion holds replicas of these objects, or whether or not the

Page 5

Figure 4. Event distribution between applications: Arrows show event movement.
Figure 5. Application joined to two channels:
Arrows show event movement.
application still receives events related to these objects.
Keeping these situations in mind, we have developed
channels. The term has many uses in many systems, but
in our system, a channel is a set of related objects. It is
implemented as an instance of an event transporter and a
multicast group designated for that transporter. An appli-
cation can join an arbitrary number of channels and create
new ones, until all available multicast groups are used up.
Figure 5 shows a single application using two channels.
One example of a channel is a set of objects in a certain
spatial area. As users move from location to location, they
can join and leave channels based on spatial areas. Another
example is the set of hazardous objects; while in the previ-
ous example, the application would replicate most objects
nearby, the hazardous objects channel could cover a larger
area, but only include those hazards. BARS also has sev-
eral interaction modules that produce many objects as they
are used. One interactor is for construction in which users
can collaborate to place points and build objects from those
points [1]; in this case, the intermediate points would be
in a channel only joined by the constructing users so that
other users would only see the final objects. Another inter-
actor allows a user to draw digital ink for interpretation by
a multimodal interaction system — this ink is turned into
new objects or user interface commands. In this case, the
user drawing the ink and the application to interface with
the multimodal system would place the ink in a separate
channel so that other users would not see these sketches out
of context.
3.4. Bridges
As we alluded to in the previous section with the multi-
modal interaction example, some of our applications com-
municate with other systems. We call these applications
bridges. These applications join both the BARS distribution
system and an outside system. They translate object cre-
ation and changes between BARS and outside systems. By
maintaining maps between BARS objects and these outside
objects, we can represent the outside objects in BARS and
vice-versa. Two systems with which we can communicate
are the Columbia Mobile Augmented Reality System [7]
and the Oregon Graduate Institute’s Quickset multimodal
interface [14].
We also mentioned that a TCP/IP transporter is avail-