Augmented Reality: Technologies, Applications, and Limitations

Abstract

In the near future we may enrich our perception of reality through revolutionary virtual augmentation. Augmented reality (AR) technologies offer an enhanced perception to help us see, hear, and feel our environments in new and enriched ways that will benefit us in fields such as education, maintenance, design, reconnaissance, to name but a few. This essay describes the field of AR, including its definition, its development history, its enabling technologies and the technological problems that developers need to overcome. To give an idea of the state of the art, some recent applications of AR technology are also discussed as well as a number of known limitation regarding human factors in the use of AR systems.

Full text

Augmented Reality: Technologies, Applications, and Limitations
D.W.F. van Krevelen∗
krevelen@cs.vu.nl
Vrije Universiteit Amsterdam, Department of Computer Science
De Boelelaan 1081a, 1081 HV Amsterdam, The Netherlands
April 18, 2007
In the near future we may enrich our perception of re-
ality through revolutionary virtual augmentation. Aug-
mented reality (AR) technologies offer an enhanced per-
ception to help us see, hear, and feel our environments
in new and enriched ways that will benefit us in fields
such as education, maintenance, design, reconnaissance,
to name but a few. This essay describes the field of AR,
including its definition, its development history, its en-
abling technologies and the technological problems that
developers need to overcome. To give an idea of the state
of the art, some recent applications of AR technology are
also discussed as well as a number of known limitations
regarding human factors in the use of AR systems.
1 Introduction
Imagine a technology with which you could see more
than others see, hear more than others hear, and perhaps
even touch, smell and taste things that others can not.
What if we had technology to perceive completely com-
putational elements and objects within our real world ex-
perience, entire creatures and structures even that help
us in our daily activities, while interacting almost uncon-
sciously through mere gestures and speech?
With such technology, mechanics could see instruc-
tions what to do next when repairing an unknown piece
of equipment, surgeons could see ultrasound scans of or-
gans while performing surgery on them, firefighters could
see building layouts to avoid otherwise invisible hazards,
soldiers could see positions of enemy snipers spotted by
unmanned reconnaissance aircraft, and we could read re-
views for each restaurant in the street we’re walking in,
or battle 10-foot tall aliens on the way to work (Feiner,
2002).
∗This essay is an effort to explore the field of augmented reality beyond
the summary presented in the course material (Dix et al., 2004, pp.
736-7). The author wishes to thank the course lecturer, Bert Bongers,
for his feedback and shared insights.
Augmented reality (AR) is this technology to cre-
ate a “next generation, reality-based interface” (Jacob,
2006) and in fact already exists, moving from labora-
tories around the world into various industries and con-
sumer markets. AR supplements the real world with vir-
tual (computer-generated) objects that appear to coexist
in the same space as the real world. According to Tech-
nology Review1, AR is recognised at MIT as one of ten
emerging technologies of 2007, and we are at the verge
of embracing this very new and exciting kind of human-
computer interaction.
For anyone who is interested and wants to get ac-
quainted with the field, this essay provides an overview
of important technologies, applications and limitations of
AR systems. First, a short definition and history of AR
are given below. Section 2 describes technologies that en-
able an augmented reality experience. To give an idea of
the possibilities of AR systems, a number of recent appli-
cations are discussed in Section 3. In Section 4 a number
of technological challenges and limitations regarding hu-
man factors are discussed that AR system designers have
to take into consideration. Finally, the essay concludes
in Section 5 with a number of directions that the author
envisions AR research might take.
Figure 1: Reality-virtuality continuum, adapted from
Azuma et al. (2001).
1http://www.techreview.com/special/emerging/
1

2 Enabling technologies
1.1 Definition
On the reality-virtuality continuum by Milgram
and Kishino (1994) (Fig. 1), AR is one part of the gen-
eral area of mixed reality. Both virtual environments (or
virtual reality) and augmented virtuality, in which real
objects are added to virtual ones, replace the surrounding
environment by a virtual one. In contrast, AR takes place
in the real world. Following the definitions in (Azuma,
1997; Azuma et al., 2001), an AR system:
•combines real and virtual objects in a real environ-
ment;
•registers (aligns) real and virtual objects with each
other; and
•runs interactively, in three dimensions, and in real
time.
Three aspects of this definition are important to men-
tion. Firstly, it is not restricted to particular display tech-
nologies such as a head-mounted display (HMD). Nor is
the definition limited to the sense of sight, as AR can
and potentially will apply to all senses, including hear-
ing, touch, and smell. Finally, removing real objects by
overlaying virtual ones, approaches known as mediated
or diminished reality, is also considered AR.
Figure 2: The world’s first head-mounted display with
the “Sword of Damocles” (Sutherland, 1968).
1.2 Brief history
The first AR prototypes (Fig. 2), created by computer
graphics pioneer Ivan Sutherland and his students at Har-
vard University and the University of Utah, appeared in
the 1960s and used a see-through HMD2to present 3D
graphics (Sutherland, 1968).
A small group of researchers at U.S. Air Force’s Arm-
strong Laboratory, the NASA Ames Research Center, the
Massachusetts Institute of Technology, and the Univer-
sity of North Carolina at Chapel Hill continued research
2http://www.telemusic.org:8080/ramgen/realvideo/HeadMounted.rm
during the 1970s and 1980s. During this time mobile de-
vices like the Sony Walkman (1979), digital watches and
personal digital organisers were introduced. This paved
the way for wearable computing (Mann, 1997; Starner
et al., 1997) in the 1990s as personal computers became
small enough to be worn at all times. Early palmtop com-
puters include the Psion I (1984), the Apple Newton Mes-
sagePad (1993), and the Palm Pilot (1996). Today, many
mobile platforms exist that may support AR, such as per-
sonal digital assistants (PDAs), tablet PCs, and mobile
phones.
It took until the early 1990s before the term ‘aug-
mented reality’ was coined by Caudell and Mizell (1992),
scientists at Boeing Corporation who were developing
an experimental AR system to help workers put together
wiring harnesses. True mobile AR was still out of reach,
but a few years later Loomis et al. (1993) developed a
GPS-based outdoor system that presents navigational as-
sistance to the visually impaired with spatial audio over-
lays. Soon computing and tracking devices became suf-
ficiently powerful and small enough to support graphical
overlay in mobile settings. Feiner et al. (1997) created an
early prototype of a mobile AR system (MARS) that reg-
isters 3D graphical tour guide information with buildings
and artifacts the visitor sees.
By the late 1990s, as AR became a research field
of its own, several conferences on AR began, includ-
ing the International Workshop and Symposium on Aug-
mented Reality, the International Symposium on Mixed
Reality, and the Designing Augmented Reality Environ-
ments workshop. Around this time, well-funded organi-
sations were formed such as the Mixed Reality Systems
Laboratory3(MRLab) in Japan and the Arvika consor-
tium4in Germany. Also, it became possible to rapidly
build AR applications thanks to freely available soft-
ware toolkits like the ARToolKit. In the mean time, sev-
eral surveys appeared that give an overview on AR ad-
vances, describe its problems, and summarise develop-
ments (Azuma, 1997; Azuma et al., 2001). By 2001,
MRLab finished their pilot research, and the symposia
were united in the International Symposium on Mixed
and Augmented Reality5(ISMAR), which has become
the major symposium for industry and research to ex-
change problems and solutions.
2 Enabling technologies
The technological demands for AR are much higher than
for virtual environments or VR, which is why the field of
AR took longer to mature than that of VR. However, the
3http://www.mr-system.com/
4http://www.arvika.de/
5http://www.augmented-reality.org/
2

2.1 Displays
key components needed to build an AR system have re-
mained the same since Ivan Sutherland’s pioneering work
of the 1960s. Displays, trackers, and graphics comput-
ers and software remain essential in many AR experi-
ences. Following the definition of AR step by step, this
section first describes display technologies that combine
the real and virtual worlds, followed by sensors and ap-
proaches to track user position and orientation for correct
registration of the virtual with the real, and user interface
technologies that allow real-time, 3D interaction. Finally
some remaining AR requirements are discussed.
2.1 Displays
Of all modalities in human sensory input, sight, sound
and/or touch are currently the senses that AR systems
commonly apply. This section mainly focuses on visual
displays, however aural (sound) displays are mentioned
briefly below. Haptic (touch) displays are discussed with
the interfaces in Section 2.3, while olfactory (smell) and
gustatory (taste) displays are less developed or practically
non-existent and will not be discussed in this essay.
2.1.1 Aural display
Aural display application in AR is mostly limited
to self-explanatory mono (0-dimensional), stereo (1-
dimensional) or surround (2-dimensional) headphones
and loudspeakers. True 3D aural display is currently
found in more immersive simulations of virtual environ-
ments and augmented virtuality or still in experimental
stages.
Haptic audio refers to sound that is felt rather than
heard (Hughes et al., 2006) and is already applied in con-
sumer devices such as Turtle Beach’s Ear Force6head-
phones to increase the sense of realism and impact, but
also to enhance user interfaces of e.g. mobile phones
(Chang and O’Sullivan, 2005). Recent developments
in this area are presented in workshops such as the in-
ternational workshop on Haptic Audio Visual Environ-
ments7and the upcoming second international workshop
on Haptic and Audio Interaction Design8.
2.1.2 Visual display
There are basically three ways to visually present an aug-
mented reality. Closest to virtual reality is video see-
through, where the virtual environment is replaced by a
video feed of reality and the AR is overlayed upon the
digitised images. Another way that includes Sutherland’s
approach is optical see-through and leaves the real-world
6http://www.turtlebeach.com/site/products/earforce/x2/
7http://www.discover.uottawa.ca/have2006/
8http://www.haid2007.org/
Figure 3: Visual display techniques and positioning
(Bimber and Raskar, 2005a).
perception alone but displays only the AR overlay by
means of transparent mirrors and lenses. The third ap-
proach is to project the AR overlay onto real objects
themselves resulting in projective displays. The three
techniques may be applied at varying distance from the
viewer (Fig. 3). Each combination of technique and dis-
tance is listed in the overview presented in Table 1 with a
comparison of their individual advantages.
Video see-through Besides being the cheapest and
easiest to implement, this display technique offers the
following advantages. Since reality is digitised, it is eas-
ier to mediate or remove objects from reality. This in-
clude removing or replacing fiducial markers or place-
holders with virtual objects (see for instance Fig. 11 and
28). Also, brightness and contrast of virtual objects are
matched easily with the real environment. The digitised
images allow tracking of head movement for better reg-
istration. It also becomes possible to match perception
delays of the real and virtual. Disadvantages of video
see-through include a low resolution of reality, a limited
field-of-view (although this can easily be increased), and
user disorientation due to a parallax (eye-offset) due to
the camera’s positioning at a distance from the viewer’s
true eye location, causing significant adjustment effort
for the viewer (Biocca and Rolland, 1998). This prob-
lem was solved at the MR Lab by aligning the video cap-
ture (Takagi et al., 2000). A final drawback is the focus
distance of this technique which is fixed in most display
types, providing poor eye accommodation. Some head-
mounted setups can however move the display (or a lens
in front of it) to cover a range of .25 meters to infinity
within .3 seconds (Sugihara and Miyasato, 1998).
3

2 Enabling technologies
(a)
(b)
Figure 4: Mobile optical see-through setups (Bimber
and Raskar, 2005a).
Optical see-through These displays not only leave the
real-world resolution in tact, they also have the advan-
tage of being cheaper, safer, and parallax-free (no eye-
offset due to camera positioning). Optical techniques
are safer because users can still see when power fails,
making this an ideal technique for military and medical
purposes. However, other input devices such as cameras
are required for interaction and registration. Also, com-
bining the virtual objects holographically through trans-
parent mirrors and lenses creates disadvantages as it re-
duces brightness and contrast of both the images and the
real-world perception, making this technique less suited
for outdoor use. The all-important field-of-view is lim-
ited for this technique and may cause clipping of vir-
tual images at the edges of the mirrors or lenses. Fi-
nally, occlusion (or mediation) of real objects is diffi-
cult because their light is always combined with the vir-
tual image. Kiyoshi Kiyokawa solved this problem for
head-worn displays by adding an opaque overlay using
an LCD panel with pixels that opacify areas to be oc-
cluded (Kiyokawa et al., 2003).
As shown in Figures 4 and 5, optical see-through tech-
niques with beam-splitting holographic optical elements
(HOEs) may be applied in head-worn displays (Fig. 4a),
hand-held displays (Fig. 4b), and spatial setups where
the AR overlay is mirrored from a planar screen (Fig. 5a)
or through a curved screen (Fig. 5b).
Virtual retinal displays or retinal scanning displays
(RSDs) solve the problems of low brightness and low
(a)
(b)
Figure 5: Spatial optical see-through setups (Bimber
and Raskar, 2005a).
field-of-view in (head-worn) optical see-through dis-
plays. A low-power laser draws a virtual image directly
onto the retina which yields high brightness and a wide
field-of-view (Fig. 6). RSD quality is not limited by the
size of pixels but only by diffraction and aberrations in
the light source, making (very) high resolutions possi-
ble as well. In postgraduate course material, Fiambo-
lis (1999) provides further information on RSD technol-
ogy. Together with their low power consumption these
displays are well-suited for extended outdoor use. Still
under development at Washington University and funded
by MicroVision9and the U.S. military, current RSDs are
mostly monochrome (red only) and monocular (single-
eye) displays. Schowengerdt et al. (2003) already devel-
oped a full-colour, binocular version with dynamic refo-
cus to accommodate the eyes (Fig. 7) that is promised to
be low-cost and light-weight.
9http://www.microvision.com
4

2.1 Displays
Figure 6: Head-worn retinal scanning display (Bimber
and Raskar, 2005a).
Figure 7: Binocular (stereoscopic) vision (Schowengerdt
et al., 2003).
Projective These displays have the advantage that they
do not require special eye-wear thus accommodating
user’s eyes during focusing, and they can cover large sur-
faces for a wide field-of-view. Projection surfaces may
range from flat, plain coloured walls to complex scale
models (Bimber and Raskar, 2005b). However, as with
optical see-through displays, other input devices are re-
quired for (indirect) interaction. Also, projectors need
to be calibrated each time the environment or the dis-
tance to the projection surface changes (crucial in mo-
bile setups). Fortunately, calibration may be automated
using cameras in e.g. a multi-walled Cave automatic vir-
tual environment (CAVE) with irregular surfaces (Raskar
et al., 1999). Furthermore, this type of display is lim-
ited to indoor use only due to low brightness and con-
trast of the projected images. Occlusion or mediation of
objects is also quite poor, but for head-worn projectors
this may be improved by covering surfaces with retro-
(b)
(a) (c)
Figure 8: Reflection types and mobile projective setups
(Bimber and Raskar, 2005a).
reflective material. Objects and instruments covered in
this material will reflect the projection directly towards
the light source which is close to the viewer’s eyes, thus
not interfering with the projection. From top to bottom,
Fig. 8a shows how light reflects from three different sur-
faces: a mirror from which light is bounced or reflected, a
normal (non-black) surface from which light is spread or
diffused, and a coated surface from which light bounces
right back to its source or retro-reflected like a car-light
reflector. Next to these surfaces are two projective AR
displays: head-worn (Fig. 8b) and handheld (Fig. 8c) in
the center. A spatial setup is shown in Figure 9.
2.1.3 Display positioning
AR displays may be classified into three categories based
on their position between the viewer and the real environ-
ment: head-worn, hand-held, and spatial (see Fig. 3).
Head-worn Visual displays attached to the head in-
clude the video/optical see-through head-mounted dis-
play (HMD), virtual retinal display (VRD), and head-
mounted projective display (HMPD). Cakmakci and Rol-
land (2006) give a recent detailed review of head-worn
display technology. A current drawback of head-worn
displays is the fact that they have to connect to graphics
computers like laptops that restrict mobility due to lim-
ited battery life. Battery life may be extended by moving
computation to distant locations and provide (wireless)
connections using standards such as IEEE 802.11b/g or
5

2 Enabling technologies
Figure 9: Spatial projective setup (Bimber and Raskar,
2005a).
BlueTooth.
Fig. 10 shows examples of four (parallax-free) head-
worn display types: Canon’s Co-Optical Axis See-
through Augmented Reality (COASTAR) video see-
through display (Tamura et al., 2001) (Fig. 10a), Konica
Minolta’s holographic optical see-through ‘Forgettable
Display’ prototype (Kasai et al., 2000) (Fig. 10b), Micro-
Vision’s monochromatic and monocular Nomad retinal
scanning display (H¨
ollerer and Feiner, 2004) (Fig. 10c),
and an organic light-emitting diode (OLED) based head-
mounted projective display (Rolland et al., 2005) (Fig.
10d).
Hand-held This category includes hand-held
video/optical see-through displays as well as hand-
held projectors. Although this category of displays is
bulkier than head-worn displays, it is currently the best
work-around to introduce AR to a mass market due
to low production costs and ease of use. For instance,
hand-held video see-through AR acting as magnifying
glasses may be based on existing consumer products like
mobile phones (M¨
ohring et al., 2004) (Fig. 11a) that
show 3D objects, or personal digital assistants/PDAs
(Wagner and Schmalstieg, 2003) (Fig. 11b) with e.g.
navigation information. Stetten et al. (2001) apply
optical see-through in their hand-held ‘sonic flashlight’
to display medical ultrasound imaging directly over the
scanned organ (Fig. 12a). One example of a hand-held
projective display or ‘AR flashlight’ is the ‘iLamp’ by
Raskar et al. (2003). This context-aware or tracked
projector adjusts the imagery based on the current
orientation of the projector relative to the environment
(Fig. 12b). Recently, MicroVision (from the retinal
Courtesy: Canon/MR Lab Courtesy: MicroVision
(a) (b)
Courtesy: Konica Minolta
(c) (d)
Figure 10: Head-worn visual displays
displays) introduced the small Pico Projector (PicoP)
which is 8mm thick, provides full-colour imagery of
1366×1024 pixels at 60Hz using three lasers, and will
probably appear embedded in mobile phones soon.
Spatial The last category of displays are placed stat-
ically within the environment and include screen-based
video see-through displays, spatial optical see-through
displays, and projective displays. These techniques lend
themselves well for large presentations and exhibitions
with limited interaction. Early ways of creating AR are
based on conventional screens (computer or television)
that show a camera feed with an AR overlay. This tech-
nique is now being applied in the world of sports tele-
vision where environments such as swimming pools and
race tracks are well defined and easy to augment. Head-
up displays (HUDs) in military cockpits are a form of
spatial optical see-through and are becoming a standard
extension for production cars to project navigational di-
rections in the windshield (Narzt et al., 2006). User view-
points relative to the AR overlay hardly change in these
cases due to the confined space. Spatial see-through dis-
plays may however appear misaligned when users move
around in open spaces, for instance when AR overlay is
presented on a transparent screen such as the ‘invisible
interface’ by Ogi et al. (2001) (Fig. 13a). 3D holographs
solve the alignment problem, as Goebbels et al. (2001)
show with the ARSyS TriCorder10 (Fig. 13b) by the Ger-
10http://www.arsys-tricorder.de
6

2.2 Tracking sensors and approaches
Positioning Head-worn Hand-held Spatial
Technique Retinal Optical Video Projective All Video Optical Projective
Mobile + + + + + − − −
Outdoor use +± ± − ± − − −
Interaction + + + + + Remote − −
Multi-user + + + + + −Limited Limited
Brightness +−+Limited + + Limited Limited
Contrast +−+Limited + + Limited Limited
Resolution Growing Growing Growing Growing Limited Limited + +
Field-of-view Growing Limited Limited Growing Limited Limited + +
Full-colour + + + + + + + +
Stereoscopic + + + + − − + +
Dynamic refocus (eye
strain)
+− − +− − + +
Occlusion ± ± +Limited ±+Limited Limited
Power economy +− − − − − − −
Opportunities Future
dominance
Current dominance Realistic,
mass-
market
Cheap, off-
the-shelf
Tuning, ergonomy
Drawbacks Tuning,
tracking
Delays Retro-
reflective
material
Processor,
memory
limits
No see-
through
metaphor
Clipping Clipping,
shadows
Table 1: Characteristics of surveyed visual AR displays.
man Fraunhofer IMK (now IAIS11) research center.
2.2 Tracking sensors and approaches
Before an AR system can display virtual objects into a
real environment, the system must be able to sense the
environment and track the viewer’s (relative) movement
preferably with six degrees of freedom (6DOF): three
variables (x, y, an z) for position and three angles (yaw,
pitch, and roll) for orientation.
There must be some model of the environment to allow
tracking for correct AR registration. Furthermore, most
environments have to be prepared before an AR system is
able to track 6DOF movement, but not all tracking tech-
niques work in all environments. To this day, determining
the orientation of a user is still a complex problem with
no single best solution.
2.2.1 Modelling environments
Both tracking and registration techniques rely on envi-
ronmental models, often 3D geometrical models. To an-
notate for instance windows, entrances, or rooms, an AR
system needs to know where they are located with regard
to the user’s current position and field of view.
Sometimes the annotations themselves may be oc-
cluded based on environmental model. For instance when
an annotated building is occluded by other objects, the
11http://www.iais.fraunhofer.de/
annotation should point to the non-occluded parts only
(Bell et al., 2001).
Fortunately, most environmental models do not need
to be very detailed about textures or materials. Usually
a “cloud” of unconnected 3D sample points suffices for
example to present occluded buildings and essentially let
users see through walls. To create a traveller guidance
service (TGS), Kim et al. (2006) used models from a ge-
ographical information system (GIS).
Stoakley et al. (1995) present users with the spatial
model itself, an oriented map of the environment or world
in miniature (WIM), to assist in navigation.
Modelling techniques Creating 3D models of large
environments is a research challenge in its own right. Au-
tomatic, semiautomatic, and manual techniques can be
employed, and Piekarski and Thomas (2001) even em-
ployed AR itself for modelling purposes. Conversely, a
laser range finder used for environmental modelling may
also enable users themselves to place notes into the envi-
ronment (Patel et al., 2006).
There are significant research problems involved in
both the modelling of arbitrary complex 3D spatial mod-
els as well as the organisation of storage and querying of
such data in spatial databases. These databases may also
need to change quite rapidly as real environments are of-
ten also dynamic.
7

2 Enabling technologies
(a)
(b)
Figure 11: Hand-held video see-through displays
2.2.2 User movement tracking
Compared to virtual environments, AR tracking devices
must have higher accuracy, a wider input variety and
bandwidth, and longer ranges (Azuma, 1997). Registra-
tion accuracy depends not only on the geometrical model
but also on the distance of the objects to be annotated.
The further away an object (i) the less impact errors in
position tracking have and (ii) the more impact errors in
orientation tracking have on the overall misregistration
(Azuma, 1999).
Tracking is usually easier in indoor settings than in
outdoor settings as the tracking devices do not have to
be completely mobile and wearable or deal with shock,
abuse, weather, etc. In stead the indoor environment
is easily modelled and prepared, and conditions such
as lighting and temperature may be controlled. Cur-
rently, unprepared outdoor environments still pose track-
ing problems with no single best solution.
Mechanical, ultrasonic, and magnetic Early tracking
techniques are restricted to indoor use as they require
special equipment to be placed around the user. The first
HMD by Sutherland (1968) was tracked mechanically
(Fig. 2) through ceiling-mounted hardware also nick-
named the “Sword of Damocles.” Devices that send and
receive ultrasonic chirps and determine the position, i.e.
ultrasonic positioning, were already experimented with
by Sutherland (1968) and are still used today. A decade
or so later Polhemus’ magnetic trackers that measure dis-
tances within electromagnetic fields were introduced by
(a)
(b)
Figure 12: Hand-held optical and projective displays
Raab et al. (1979). These are also still in use today and
had much more impact on VR and AR research.
Global positioning systems For outdoor tracking by
global positioning system (GPS) there exist the American
24-satellite Navstar GPS (Getting, 1993), the Russian
counterpart constellation Glonass, and the 30-satellite
GPS Galileo, currently being launched by the European
Union and operational in 2010.
Direct visibility with at least four satellites is no longer
necessary with assisted GPS12 (A-GPS), a worldwide
network of servers and base stations enable signal broad-
cast in for instance urban canyons and indoor environ-
ments. Plain GPS is accurate to about 10-15 meters, but
with the wide area augmentation system (WAAS) tech-
nology may be increased to 3-4 meters. For more accu-
racy, the environments have to be prepared with a local
base station that sends a differential error-correction sig-
nal to the roaming unit: differential GPS yields 1-3 me-
ter accuracy, while the real-time-kinematic or RTK GPS,
based on carrier-phase ambiguity resolution, can estimate
positions accurately to within centimeters. Update rates
of commercial GPS systems such as the MS750 RTK re-
ceiver by Trimble13 have increased from five to twenty
times a second and are deemed suitable for tracking fast
motion of people and objects (H¨
ollerer and Feiner, 2004).
Radio Other tracking methods that require environ-
ment preparation by placing devices are based on ultra
wide band radio waves. Active radio frequency iden-
12http://www.globallocate.com/
13http://www.trimble.com/
8

2.2 Tracking sensors and approaches
(a)
Courtesy: Fraunhofer IMK/IAIS
(b)
Figure 13: Spatial visual displays
Figure 14: An ultra wide band (UWB) based positioning
system c
2007 PanGo.
tification (RFID) chips may be positioned inside struc-
tures such as aircraft (Willers, 2006) to allow in situ po-
sitioning. Complementary to RFID one can apply the
wide-area IEEE 802.11b/g standards for both wireless
networking and tracking as well. The achievable reso-
lution depends on the density of deployed access points
in the network. Several techniques are researched by
Bahl and Padmanabhan (2000); Castro et al. (2001) and
vendors like PanGo14 (Fig. 14), AeroScout15 and Eka-
hau16 offer integrated systems for personnel and equip-
ment tracking in for instance hospitals.
14http://www.pango.com/
15http://aeroscout.com/
16http://www.ekahau.com/
Inertial Accelerometers and gyroscopes are sourceless
inertial sensors, usually part of hybrid tracking sys-
tems, that do not require prepared environments. Timed
measurements can provide a practical dead-reckoning
method to estimate position when combined with ac-
curate heading information. To minimise errors due to
drift, the estimates must periodically be updated with
accurate measurements. The act of taking a step can
also be measured, i.e. they can function as pedometers.
Currently micro-electromechanical (MEM) accelerome-
ters and gyroscopes are already making their way into
mobile phones (e.g. Samsung’s SCH-S310 and Sharp’s
V603SH) to allow ‘writing’ of phone numbers in the air,
etc.
Optical Promising approaches for 6DOF pose estima-
tion of users and objects in general settings are vision-
based. In closed-loop tracking, the field of view of the
camera coincides with that of the user (e.g. in video see-
through) allowing for pixel-perfect registration of virtual
objects. Conversely in open-loop tracking, the system
relies only on the sensed pose of the user and the envi-
ronmental model.
Using one or two tiny cameras, model-based ap-
proaches can recognise landmarks (given an accurate en-
vironmental model) or detect relative movement dynam-
ically between frames. There are a number of techniques
to detect scene geometry (e.g. landmark or template
matching) and camera motion in both 2D (e.g. optical
flow) and 3D which require varying amounts of com-
putation. Julier and Bishop (2002) combine some in a
number of test scenarios. Although early vision-based
tracking and interaction applications in prepared envi-
ronments use fiducial markers (e.g. Naimark and Foxlin,
2002; Schmalstieg et al., 2000) or light emitting diodes
(LEDs) to see how and where to register virtual objects,
there is a growing body of research on ‘markerless AR’
for tracking physical positions (Chia et al., 2002; Com-
port et al., 2003; Ferrari et al., 2001; Gordon and Lowe,
2004; Koch et al., 2005). Robustness is still low and
computational costs high, but results of these pure vision-
based approaches (hybrid and/or markerless) for general-
case, real-time tracking are very promising.
Hybrid Commercial hybrid tracking systems became
available during the 1990s and use for instance elec-
tromagnetic compasses (magnetometers), gravitational
tilt sensors (inclinometers), and gyroscopes (mechanical
and optical) for orientation tracking and ultrasonic, mag-
netic, and optical position tracking. Hybrid tracking ap-
proaches are currently the most promising way to deal
with the difficulties posed by general indoor and outdoor
mobile AR environments (H¨
ollerer and Feiner, 2004).
9

2 Enabling technologies
Azuma et al. (2006) investigate hybrid methods without
vision-based tracking suitable for military use at night
in an outdoor environment with less than ten beacons
mounted on for instance unmanned air vehicles (UAVs).
2.3 User interface and interaction
Besides registering virtual data with the user’s real world
perception, the system needs to provide some kind of in-
terface with both virtual and real objects.
New UI paradigm WIMP (windows, icons, menus, and
pointing), as the conventional desktop UI metaphor is re-
ferred to, does not apply that well to AR systems. Not
only is interaction required with six degrees of freedom
(6DOF) rather than 2D, the use of conventional devices
like a mouse and keyboard are cumbersome to wear and
reduce the AR experience.
Like in WIMP UIs, AR interfaces have to support se-
lecting, positioning, and rotating of virtual objects, draw-
ing paths or trajectories, assigning quantitative values
(quantification) and text input. However as a general UI
principle, AR interaction also includes the selection, an-
notation, and, possibly, direct manipulation of physical
objects. This computing paradigm is still a challenge
(H¨
ollerer and Feiner, 2004).
Figure 15: StudierStube’s general-purpose Personal In-
teraction Panel with 2D and 3D widgets and
a 6DOF pen (Schmalstieg et al., 2000).
Tangible UI and 3D pointing Early mobile AR systems
simply use mobile trackballs, trackpads and gyroscopic
mice to support continuous 2D pointing tasks. This is
largely because the systems still use a WIMP interface
and accurate gesturing to WIMP menus would otherwise
require well-tuned motor skills from the users. Ideally
the number of extra devices that have to be carried around
in mobile UIs is reduced, but this may be difficult with
current mobile computing and UI technologies.
Devices like the mouse are tangible and unidirec-
tional, they communicate from the user to the AR system
only. Common 3D equivalents are tangible user inter-
faces (TUIs) like paddles and wands. Ishii and Ullmer
(1997) discuss a number of tangible interfaces devel-
oped at MIT’s Tangible Media Group17 including ph-
icons (physical icons) and sliding instruments. Some
TUIs have placeholders or markers on them so the AR
system can replace them visually with virtual objects.
Poupyrev et al. (2001) use tiles with fiducial markers,
while in StudierStube, Schmalstieg et al. (2000) allow
users to interact through a Personal Interaction Panel with
2D and 3D widgets that also recognises pen-based ges-
tures in 6DOF (Fig. 15).
Figure 16: SensAble’s PHANTOM Premium 3.0 6DOF
haptic device.
Haptic UI and gesture recognition TUIs with
bi-directional, programmable communication through
touch are called haptic UIs. Haptics is like teleoperation,
but the remote slave system is purely computational, i.e.
“virtual.” Haptic devices are in effect robots with a single
task: to interact with humans (Hayward et al., 2004).
The haptic sense is divided into the kinaesthetic sense
(force, motion) and the tactile sense (tact, touch). Force
feedback devices like joysticks and steering wheels can
suggest impact or resistance and are well-known among
gamers. A popular 6DOF haptic device in teleoperation
and other areas is the PHANTOM (Fig. 16). It option-
ally provides 7DOF interaction through a pinch or scis-
sors extension. Tactile feedback devices convey parame-
ters such as roughness, rigidity, and temperature. Benali-
17http://tangible.media.mit.edu/
10

2.3 User interface and interaction
Khoudja et al. (2004) survey tactile interfaces used in
teleoperation, 3D surface simulation, games, etc.
Figure 17: SensAble’s CyberTouch tactile data glove
provides pulses and vibrations to each finger.
Data gloves use diverse technologies to sense and actu-
ate and are very reliable, flexible and widely used in VR
for gesture recognition. In AR however they are suitable
only for brief, casual use, as they impede the use of hands
in real world activities and are somewhat awkward look-
ing for general application. Buchmann et al. (2004) con-
nected buzzers to the fingertips informing users whether
they are ‘touching’ a virtual object correctly for manip-
ulation, much like the CyberGlove with CyberTouch by
SensAble18 (Fig. 17).
Visual UI and gesture recognition In stead of using
hand-worn trackers, hand movement may also be tracked
visually, leaving the hands unencumbered. A head-worn
or collar-mounted camera pointed at the user’s hands can
be used for gesture recognition. Through gesture recog-
nition, an AR could automatically draw up reports of
activities (Mayol and Murray, 2005). For 3D interac-
tion, UbiHand19 uses wrist-mounted cameras enable ges-
ture recognition (Ahmad and Musilek, 2006), while the
Mobile Augmented Reality Interface Sign Interpretation
Language20 (Antoniac and Pulli, 2001) recognises hand
gestures on a virtual keyboard displayed on the user’s
hand (Fig. 18).
Cameras are also useful to record and document the
user’s view, e.g. for providing a live video feed for tele-
conferencing, for informing a remote expert about the
findings of AR field-workers, or simply for document-
18http://www.sensable.com/
19http://www.ece.ualberta.ca/ fahmad
20http://marisil.org/
ing and storing everything that is taking place in front of
the mobile AR system user.
Common in indoor virtual or augmented environments
is the use of additional orientation and position track-
ers to provide 6DOF hand tracking for manipulating vir-
tual objects. For outdoor environments, Foxlin and Har-
rington (2000) experimented with ultrasonic tracking of
finger-worn acoustic emitters using three head-worn mi-
crophones.
Gaze tracking Using tiny cameras to observe user
pupils and determine the direction of their gaze is a tech-
nology with potential for AR. The difficulties are that it
needs be incorporated into the eye-wear, calibrated to the
user to filter out involuntary eye movement, and posi-
tioned at a fixed distance. With enough error correction,
gaze tracking alternatives for the mouse such as Stan-
ford’s EyePoint21 (Kumar and Winograd, 2007) provides
a dynamic history of user’s interests and intentions that
may help the UI adapt to the future contexts.
Aural UI and speech recognition To reach the ideal
of an inconspicuous UI, auditory UIs may become an
important part of the solution. Microphones and ear-
phones are easily hidden and allow auditory UIs to deal
with speech recognition, speech recording for human-to-
human interaction, audio information presentation, and
audio dialogue. Although noisy environments pose prob-
lems, audio can be valuable in multimodal and multime-
dia UIs.
Text input Achieving fast and reliable text input to a
mobile computer remains hard. Standard keyboards re-
quire much space and a flat surface, and the current com-
mercial options such as small, foldable, inflatable, or
laser-projected virtual keyboards are cumbersome, while
soft keyboards take up valuable screen space. Popular
choices in the mobile community include chording key-
boards such as the Twiddler2 by Handykey22 that re-
quire key combinations to encode a single character. Of
course mobile AR systems based on hand-held devices
like tablet PCs, PDAs or mobile phones already support
alphanumeric input through keypads or pen-based hand-
writing recognition (facilitated by e.g. dictionaries or
shape writing technologies), but this cannot be applied
in all situations. Glove-based and vision-based hand ges-
ture tracking do not yet provide the ease of use and ac-
curacy for serious adoption. Speech recognition however
has improved over the years in both speed and accuracy
and, when combined with a fall-back device (e.g., pen-
based systems or special purpose chording or miniature
21http://hci.stanford.edu/research/GUIDe/
22http://www.handykey.com/
11

2 Enabling technologies
Figure 18: Mobile Augmented Reality Interface Sign Interpretation Language c
2004 Peter Antoniac.
keyboards), may be a likely candidate for providing text
input to mobile devices in a wide variety of situations
(H¨
ollerer and Feiner, 2004).
Hybrid UI With each modality having its drawbacks
and benefits, AR systems are likely to use a multimodal
UI. A synchronised combination of for instance gestures,
speech, sound, vision and haptics may provide users with
a more natural and robust, yet predictable UI.
Context awareness The display and tracking devices
discussed earlier already provide some advantages for an
AR interface. A mobile AR system is aware of the user’s
position and orientation and can adjust the UI accord-
ingly. Such context awareness can reduce UI complexity
for ecample by dealing only with virtual or real objects
that are nearby or within visual range.
Towards human-machine symbiosis Another class
of sensors gathers information about the user’s state. Bio-
metric devices can measure heart-rate and bioelectric sig-
nals, such as galvanic skin response, electroencephalo-
gram (neural activity), or electromyogram (muscle activ-
ity) data in order to monitor biological activity. Affective
computing (Picard, 1997) aims to make computers more
aware of the emotional state of their users and able to
adapt accordingly. Although the future may hold human-
machine symbioses (Licklider, 1960), current integration
of UI technology is restricted to devices that are worn
or perhaps embroidered to create computationally aware
clothes (Farringdon et al., 1999).
2.4 More AR requirements
Besides tracking, registration, and interaction, H ¨
ollerer
and Feiner (2004) mention three more requirements for
a mobile AR system: computational framework, wire-
less networking, and data storage and access technology.
Content is of course also required, so some authoring
tools are mentioned here as well.
Figure 19: Typical AR system framework tasks
Frameworks AR systems have to perform some typ-
ical tasks like tracking, sensing, display and interac-
tion (Fig. 19). These can be supported by fast pro-
totyping frameworks that are developed independently
from their applications. Easy integration of AR devices
and quick creation of user interfaces can be achieved
with frameworks like the ARToolKit23, probably the
best known and most widely used. Other frameworks
include COTERIE24,StudierStube25 (Szalav´
ari et al.,
1998), DWARF26, D’Fusion by Total Immersion27, etc.
Networks and databases AR systems usually present
a lot of knowledge to the user which is obtained through
networks. Especially mobile and collaborative AR sys-
tems will require suitable (wireless) networks to support
data retrieval and multi-user interaction over larger dis-
tances. Moving computation load to remote servers is
one approach to reduce weight and bulk of mobile AR
systems (Behringer et al., 2000; Mann, 1997). How to
get to the most relevant information with the least effort
from databases, and how to minimise information pre-
sentation are still open research questions.
Content The author believes that commercial success
of AR systems will depend heavily on the available types
of content. Scientific and industrial applications are usu-
ally based on specialised content, but presenting com-
23http://artoolkit.sourceforge.net/
24http://www.cs.columbia.edu/graphics/projects/coterie/
25http://studierstube.icg.tu-graz.ac.at/
26http://www.augmentedreality.de/
27http://www.t-immersion.com/
12

3.1 Personal information systems
mercial content to the common user will remain a chal-
lenge if AR is not applied in everyday life.
Some of the available AR authoring tools are the
CREATE tool from Information in Place28, the DART
toolkit29 and the MARS Authoring Tool30. Companies
like Thinglab31 assist in 3D scanning or digitising of ob-
jects. Optical capture systems, capture suits, and other
tracking devices available at companies like Inition32 are
tools for creating some life AR content beyond ‘simple’
annotation.
Creating or recording dynamic content could bene-
fit from techniques already developed in the movie and
games industries, but also from accessible 3D drawing
software like Google SketchUp33. Storing and replaying
user experiences is a valuable extension to MR system
functionality and are provided for instance in HyperMem
(Correia and Romero, 2006).
3 Applications
Over the years, researchers and developers find more and
more areas that could benefit from augmentation. The
first systems focused on military, industrial and medical
application, but AR systems for commercial use and en-
tertainment appeared soon after. Which of these appli-
cations will trigger wide-spread use is anybody’s guess.
This section discusses some areas of application grouped
similar to the ISMAR 2007 symposium34 categorisation.
3.1 Personal information systems
H¨
ollerer and Feiner (2004) believe one of the biggest
potential markets for AR could prove to be in personal
wearable computing. AR may serve as an advanced, im-
mediate, and more natural UI for wearable and mobile
computing in personal, daily use. For instance, AR could
integrate phone and email communication with context-
aware overlays, manage personal information related to
specific locations or people, provide navigational guid-
ance, and provide a unified control interface for all kinds
of appliances in and around the home. Such a platform
also presents direct marketing agencies with many op-
portunities to offer coupons to passing pedestrians, place
virtual billboards, show virtual prototypes, etc. With all
these different uses, AR platforms should preferably of-
fer a filter to manage what content they display.
28http://www.informationinplace.com/
29http://www.gvu.gatech.edu/dart/
30http://www.cs.columbia.edu/graphics/projects/mars/
31http://www.thinglab.co.uk/
32http://www.inition.co.uk/
33http://www.sketchup.com/
34http://www.ismar07.org/
Figure 20: Personal Awareness Assistant c
Accenture.
Personal Assistance Available from Accenture is the
Personal Awareness Assistant (Fig. 20) which automati-
cally stores names and faces of people you meet, cued by
words as ‘how do you do’. Speech recognition also pro-
vides a natural interface to retrieve the information that
was recorded earlier. Journalists, police, geographers and
archaeologists could use AR to place notes or signs in the
environment they are reporting on or working in.
(a) (b)
Figure 21: Pedestrian navigation (Narzt et al., 2003) and
traffic warning (T¨
onnis et al., 2005).
Navigation Navigation in prepared environments has
been tried and tested for some time. Rekimoto (1997)
presented NaviCam for indoor use that augmented a
video stream from a hand held camera using fiducial
markers for position tracking. Starner et al. (1997) con-
sider applications and limitations of AR for wearable
computers, including problems of finger tracking and fa-
cial recognition. Narzt et al. (2003, 2006) discuss nav-
igation paradigms for (outdoor) pedestrians (Fig. 21a)
and cars that overlay routes, highway exits, follow-me
cars, dangers, fuel prices, etc. They prototyped video
see-through PDAs and mobile phones and envision even-
13

3 Applications
tual use in car windshield heads-up displays. T¨
onnis et al.
(2005) investigate the success of using AR warnings to
direct a car driver’s attention towards danger (Fig. 21b).
Kim et al. (2006) describe how a 2D traveler guidance
service can be made 3D using GIS data for AR naviga-
tion. Nokia’s MARA project35 researches deployment of
AR on current mobile phone technology.
Touring H¨
ollerer et al. (1999) use AR to create situ-
ated documentaries about historic events, while Vlahakis
et al. (2001) present the ArcheoGuide project that recon-
structs a cultural heritage site in Olympia, Greece. With
this system, visitors can view and learn ancient architec-
ture and customs. Similar systems have been developed
for the Pompeii site (Magnenat-Thalmann and Papagian-
nakis, 2005; Papagiannakis et al., 2005). The lifeClip-
per36 project does about the same for structures and tech-
nologies in medieval Germany and is moving from an art
project to serious AR exhibition. Bartie and Mackaness
(2006) introduced a touring system to explore landmarks
in the cityscape of Edinburgh that works with speech
recognition.
3.2 Industrial and military applications
Design, assembly, and maintenance are typical areas
where AR may prove useful. These activities may be
augmented in both corporate and military settings.
(a) (b)
Figure 22: Spacedesign (Fiorentino et al., 2002) and
Clear and Present Car (Tamura, 2002; Tamura
et al., 2001).
Design Fiorentino et al. (2002) introduced the
SpaceDesign MR workspace (based on the StudierStube
framework) that allows for instance visualisation and
modification of car body curvature and engine layout
(Fig. 22a). Volkswagen intends to use AR for comparing
calculated and actual crash test imagery (Friedrich
35http://research.nokia.com/research/projects/mara/
36http://www.torpus.com/lifeclipper/
and Wohlgemuth, 2002). The MR Lab used data from
DaimlerChrysler’s cars to create Clear and Present Car,
a simulation where one can open the door of a virtual
concept car and experience the interior, dash board lay
out and interface design for usability testing (Tamura,
2002; Tamura et al., 2001). Notice how the steering
wheel is drawn around the hands, rather than over them
(Fig. 22b).
Figure 23: Robot sensor data visualisation (Collett
and MacDonald, 2006).
Another interesting application presented by Collett
and MacDonald (2006) is the visualisation of robot pro-
grams (Fig. 23). With small robots such as the auto-
mated vacuum cleaner Roomba from iRobot37 entering
our daily lives, visualising their sensor ranges and in-
tended trajectories might be welcome extensions.
Assembly BMW is experimenting with AR to improve
welding processes on their cars (Sandor and Klinker,
2005). Assisting the production process at Boeing,
Mizell (2001) use AR to overlay schematic diagrams
and accompanying documentation directly onto wooden
boards on which electrical wires are routed, bundled, and
sleeved. Curtis et al. (1998) verify the AR and find that
workers using AR create wire bundles as well as con-
ventional approaches, even though tracking and display
technologies were limited at the time.
Figure 24: Airbus water system assembly (Willers,
2006).
At EADS, supporting EuroFighter’s nose gear assem-
bly is researched (Friedrich and Wohlgemuth, 2002)
37http://www.irobot.com/
14

3.3 Medical applications
while (Willers, 2006) research AR support for Airbus’
cable and water systems (Fig. 24). Leading (and talking)
workers through the assembly process of large aircraft is
not suited for stationary AR solutions, yet mobility and
tracking with so much metal around also prove to be chal-
lenging.
An extra benefit of augmented assembly and construc-
tion is the possibility to monitor and schedule individ-
ual progress in order to manage large complex construc-
tion projects. An example by Feiner et al. (1999) gener-
ates overview renderings of the entire construction scene
while workers use their HMD to see which strut is to
be placed where in a space-frame structure. Distributed
interaction on construction is further studied by Olwal
and Feiner (2004).
Maintenance Complex machinery or structures require
a lot of skill from maintenance personnel and AR is prov-
ing useful in this area, for instance in providing “x-ray
vision” or automatically probing the environment with
extra sensors to direct the users attention to problem
sites. Klinker et al. (2001) present an AR system for
the inspection of power plants at Framatome ANP (today
AREVA). Friedrich and Wohlgemuth (2002) show the in-
tention to support electrical troubleshooting of vehicles at
Ford and according to a MicroVision employee38, Honda
and Volvo ordered Nomad Expert Vision Technician sys-
tems to assist their technicians with vehicle history and
repair information (Kaplan-Leiserson, 2004).
Combat and simulation Satellite navigation, heads-
up displays for pilots, and also much of the current
AR research at universities and corporations are the re-
sult of military funding. Companies like Information
in Place have contracts with the Army, Air Force and
Coast Guard, as land warrior and civilian use of AR may
overlap in for instance navigational support, communi-
cations enhancement, repair and maintenance and emer-
gency medicine. Extra benefits specific for military users
may be training in large-scale combat scenarios and sim-
ulating real-time enemy action, as in the Battlefield Aug-
mented Reality System (BARS) by Julier et al. (1999)
and research by Piekarski et al. (1999). Not overloading
the user with too much information is critical and is be-
ing studied by Julier et al. (2000). The BARS system also
provides tools to author the environment with new 3D
information that other system users see in turn (Baillot
et al., 2001). Azuma et al. (2006) investigate the projec-
tion of reconnaissance data from unmanned air vehicles
for land warriors.
38http://microvision.blogspot.com/2004/05/looking-up.html
3.3 Medical applications
Similar to maintenance personnel, roaming nurses and
doctors could benefit from important information being
delivered directly to their glasses (Hasvold, 2002). Sur-
geons however require very precise registration while AR
system mobility is less of an issue.
Figure 25: Simulated visualisation in laparoscopic
surgery for left and right eye (Fuchs et al.,
1998).
An early optical see-through augmentation is pre-
sented by Fuchs et al. (1998) for laparoscopic surgery39
where the overlayed view of the laparoscopes inserted
through small incisions is simulated (Fig. 25). Pietrzak
et al. (2006) confirm that the use of 3D imagery in laparo-
scopic surgery still has to be proven, but the opportunities
are well documented.
(a) (b)
Figure 26: AR overlay of a medical scan (Merten, 2007).
There are many AR approaches being tested in
medicine with live overlays of ultrasound, CT, and MR
scans. Navab et al. (1999) take advantage of the physi-
cal constraints of a C-arm x-ray machine to automatically
calibrate the cameras with the machine and register the x-
ray imagery with the real objects. Vogt et al. (2006) use
video see-through HMD to overlay MR scans on heads
and provide views of tool manipulation hidden beneath
tissue and surfaces, while Merten (2007) gives an impres-
sion of MR scans overlayed on feet (Fig. 26).
39Laparoscopic surgery uses slender camera systems (laparoscopes) and
instruments inserted in the abdomen and/or pelvis cavities through
small incisions for reduced patient recovery times.
15

3 Applications
3.4 AR for entertainment
Like VR, AR can be applied in the entertainment industry
to create AR games, but also to increase visibility of im-
portant game aspects in life sports broadcasting. In these
cases where a large public is reached, AR can also serve
advertisers to show virtual ads and product placements.
(a) (b)
Figure 27: AR in life sports broadcasting: NASCAR rac-
ing and football (Azuma et al., 2001).
Sports broadcasting Swimming pools, football fields,
race tracks and other sports environments are well-known
and easily prepared, which video see-through augmenta-
tion through tracked camera feeds easy. One example is
the FoxTrax system (Cavallaro, 1997), used to highlight
the location of a hard-to-see hockey puck as it moves
rapidly across the ice, but AR is also applied to anno-
tate racing cars (Fig. 27a), snooker ball trajectories, life
swimmer performances, etc. Thanks to predictable en-
vironments (uniformed players on a green, white, and
brown field) and chroma-keying techniques, the annota-
tions are shown on the field and not on the players (Fig.
27b).
Games Extending on a platform for military simula-
tion (Piekarski et al., 1999) based on the ARToolKit,
Thomas et al. (2000) created ‘ARQuake’ where mobile
users fight virtual enemies in a real environment. A
general purpose outdoor AR platform, ‘Tinmith-Metro’
evolved from this work and is available at the Wearable
Computer Lab40 (Piekarski and Thomas, 2001, 2002).
At a rage41 a similar platform for outdoor games such
as ‘Sky Invaders’ is available, and Cheok et al. (2002)
introduce the adventurous ‘Game-City’. Crabtree et al.
(2004); Flintham et al. (2003) discuss experiences with
mobile MR game ‘Bystander’ where virtual online play-
ers avoid capture from real-world cooperating runners.
40http://www.tinmith.net/
41http://www.a-rage.com/
(a) (b)
Figure 28: Mobile AR tennis with the phones used as
rackets (Henrysson et al., 2005).
A number of games have been developed for pre-
pared indoor environments, such as the alien-battling
‘AquaGauntlet’ (Tamura et al., 2001), dolphin-juggling
‘ContactWater’, ‘ARHockey’, and ‘2001 AR Odyssey’
(Tamura, 2002). In ‘AR-Bowling’ Matysczok et al.
(2004) study game-play, and Henrysson et al. (2005) cre-
ated AR tennis for the Nokia mobile phone (Fig. 28).
Early AR games also include AR air hockey (Ohshima
et al., 1998), collaborative combat against virtual ene-
mies (Ohshima et al., 1999), and an AR-enhanced pool
game (Jebara et al., 1997).
3.5 AR for the office
Besides in games, collaboration in office spaces is an-
other area where AR may prove useful, for example in
public management or crisis situations, urban planning,
etc.
Collaboration Having multiple people view, discuss,
and interact with 3D models simultaneously is a major
potential benefit of AR. Collaborative environments al-
low seamless integration with existing tools and prac-
tices and enhance practice by supporting remote and
collocated activities that would otherwise be impossi-
ble (Billinghurst and Kato, 1999). Benford et al. (1998)
name four examples where shared MR spaces may ap-
ply: doctors diagnosing 3D scan data, construction engi-
neers discussing plans and progress data, environmental
planners discussing geographical data and urban devel-
opment, and distributed control rooms such as Air Traffic
Control operating through a common visualisation.
Augmented Surfaces by Rekimoto and Saitoh (1999)
leaves users unencumbered but is limited to adding vir-
tual information to the projected surfaces. Examples of
collaborative AR systems using see-through displays in-
clude both those that use see-through handheld displays
(such as Transvision (Rekimoto, 1996) and MagicBook
(Billinghurst et al., 2001)) and see-through head-worn
16

(a) (b)
Figure 29: MR2(Tamura, 2002) and ARTHUR (Broll
et al., 2004).
displays (such as Emmie (Butz et al., 1999), and Studier-
Stube (Szalav´
ari et al., 1998), MR2(Tamura, 2002) and
ARTHUR (Broll et al., 2004)). Privacy management is
handled in the Emmie system through such metaphors as
lamps and mirrors. Making sure everybody knows what
someone is pointing at is a problem that StudierStube
overcomes by using virtual representation of physical
pointers. Similarly, Tamura (2002) presented a mixed
reality meeting room (MR2) for 3D presentations (Fig.
29a). For urban planning purposes, Broll et al. (2004)
introduced ARTHUR, complete with pedestrian flow vi-
sualisation (Fig. 29b) but lacking augmented pointers.
3.6 Education and training
Close to earlier mentioned collaborative applications like
games and planning are AR tools that support educa-
tion with 3D objects. Many studies research this area
of application such as (Billinghurst, 2003; G. Heide-
mann and Ritter, 2005; Hughes et al., 2005; Kommers
and Zhiming; Kritzenberger et al., 2002; Pan et al., 2006;
Winkler et al., 2003).
(a) (b)
Figure 30: Construct3D (Kaufmann et al., 2000) and
MARIE (Liarokapis et al., 2004).
Kaufmann (2002); Kaufmann et al. (2000) introduce
the Construct3D tool for math and geometry education,
based on the StudierStube framework (Fig. 30a). In
MARIE (Fig. 30b), based in turn on the Construct3D
tool, Liarokapis et al. (2004) employ screen-based AR
with Web3D to support engineering education. Recently
MIT Education Arcade introduced game-based learning
in ‘Mystery at the museum’ and ‘Environmental Detec-
tives’ where each educative game has an “engaging back-
story, differentiated character roles, reactive third parties,
guided debriefing, synthetic activities, and embedded re-
call/replay to promote both engagement and learning”
(Kaplan-Leiserson, 2004).
4 Limitations
AR faces technical challenges regarding for example
binocular (stereo) view, high resolution, colour depth, lu-
minance, contrast, field of view, and focus depth. How-
ever, before AR becomes accepted as part of user’s ev-
eryday life, just like a mobile phone and a personal dig-
ital assistant (PDA), issues regarding intuitive interfaces,
costs, weight, power usage, ergonomics, and appearance
must also be addressed. A number of limitations, some of
which have been mentioned earlier, are categorised here.
Portability and outdoor use Most mobile AR systems
mentioned in this survey are cumbersome, requiring a
heavy backpack to carry the PC, sensors, display, bat-
teries, and everything else. Connections between all the
devices must be able to withstand outdoor use, including
weather and shock, but universal serial bus (USB) con-
nectors are known to fail easily. However, recent devel-
opments in mobile technology like cell phones and PDAs
are bridging the gap towards mobile AR.
Optical and video see-through displays are usually un-
suited for outdoor use due to low brightness, contrast,
resolution, and field of view. However, recently devel-
oped at MicroVision, laser-powered displays offer a new
dimension in head-mounted and hand-held displays that
overcomes this problem.
Most portable computers have only one CPU which
limits the amount of visual and hybrid tracking. More
generally, consumer operating systems are not suited for
real-time computing, while specialised real-time operat-
ing systems don’t have the drivers to support the sensors
and graphics in modern hardware.
Tracking and (auto)calibration Tracking in unpre-
pared environments remains a challenge but hybrid ap-
proaches are becoming small enough to be added to mo-
bile phones or PDAs. Calibration of these devices is
still complicated and extensive, but this may be solved
17

References
through calibration-free or auto-calibrating approaches
that minimise set-up requirements. The latter use re-
dundant sensor information to automatically measure and
compensate for changing calibration parameters (Azuma
et al., 2001).
Latency A large source of dynamic registration errors
are system delays (Azuma et al., 2001). Techniques like
precalculation, temporal stream matching (in video see-
through such as live broadcasts), and prediction of future
viewpoints may solve some delay. System latency can
also be scheduled to reduce errors through careful system
design, and pre-rendered images may be shifted at the
last instant to compensate for pan-tilt motions. Similarly,
image warping may correct delays in 6DOF motion (both
translation and rotation).
Depth perception One difficult registration problem
is accurate depth perception. Stereoscopic displays
help, but additional problems including accommodation-
vergence conflicts or low resolution and dim displays
cause object to appear further away than they should be
(Drascic and Milgram, 1996). Correct occlusion amelio-
rates some depth problems (Rolland and Fuchs, 2000),
as does consistent registration for different eyepoint lo-
cations (Vaissie and Rolland, 2000).
Adaptation In early video see-through systems with a
parallax, users need to adapt to vertical displaced view-
points. In an experiment by Biocca and Rolland (1998),
subjects exhibit a large overshoot in a depth-pointing task
after removing the HMD.
Fatigue and eye strain Like the parallax problem,
biocular displays (where both eyes see the same image)
cause significantly more discomfort than monocular or
binocular displays, both in eye strain and fatigue (Ellis
et al., 1997).
Overload and over-reliance Aside from technical
challenges, the user interface must also follow some
guidelines as not to overload the user with informa-
tion while also preventing the user to overly rely on the
AR system such that important cues from the environ-
ment are missed (Tang et al., 2003). At BMW, Bengler
and Passaro (2004) use guidelines for AR system design
in cars, including orientation on the driving task, no mov-
ing or obstructing imagery, add only information that im-
proves driving performance, avoid side effects like tunnel
vision and cognitive capture, and only use information
that does not distract, intrude or disturb given different
situations.
Social acceptance Getting people to use AR may be
more challenging than expected, and many factors play a
role in social acceptance of AR ranging from unobtrusive
fashionable appearance (gloves, helmets, etc.) to privacy
concerns. For instance, Accenture’s Assistant (Fig. 20)
blinks a light when it records for the sole purpose of alert-
ing the person who is being recorded. These fundamental
issues must be addressed before AR is widely accepted
(H¨
ollerer and Feiner, 2004).
5 Conclusion
This essay presented a state of the art survey of AR tech-
nologies, applications and limitations. The comparative
table on displays (Table 1) and survey of frameworks and
content authoring tools (Section 2.4) are specifically new
to the AR research literature. With over a hundred ref-
erences this essay has become a comprehensive investi-
gation into AR and hopefully provides a suitable starting
point for readers new to the field.
AR has come a long way but still has some distance
to go before industries, the military and the general pub-
lic will accept it as a familiar user interface. For exam-
ple, Airbus CIMPA still struggles to get their AR systems
for assembly support accepted by the workers (Willers,
2006). On the other hand, companies like Information in
Place estimate that by 2014, 30% of mobile workers will
be using augmented reality. Within 5-10 years, Feiner
(2002) believes that “augmented reality will have a more
profound effect on the way in which we develop and in-
teract with future computers.” With the advent of such
complementary technologies as tactile networks by Red-
Tacton42, artificial intelligence, cybernetics, and (non-
invasive) brain-computer interfaces, AR might soon pave
the way for ubiquitous (anytime-anywhere) computing
(Weiser, 1991) of a more natural kind (Abowd and My-
natt, 2000) or even human-machine symbiosis as envi-
sioned by Licklider (1960).
References
ABOW D, G. D. A ND MY NATT, E. D . Charting past,
present, and future research in ubiquitous comput-
ing. ACM Trans. on Computer-Human Interaction 7,
1 (Mar. 2000), pp. 29–58. 5
AHMAD, F. AND MU SIL EK , P. A keystroke and pointer
control input interface for wearable computers. In Per-
Com’06: Proc. 4th IEEE Int’l Conf. Pervasive Com-
puting and Communications (2006), pp. 2–11. 2.3
ANTON IAC , P. AND PULL I, P. Marisil–mobile user
interface framework for virtual enterprise. In Proc.
42http://www.redtacton.com/
18

References
7th Int’l Conf. Concurrent Enterprising (Bremen, June
2001), pp. 171–180. 2.3
AZU MA , R . T. A survey of augmented reality. Presence
6, 4 (Aug. 1997), pp. 355–385. 1.1, 1.2, 2.2.2
AZU MA , R. T. Mixed Reality: Merging Real and Virtual
Worlds. Ohmsha/Springer, Tokyo/New York, 1999,
ch. The challenge of making augmented reality work
outdoors, pp. 379–390. ISBN 3-540-65623-5. 2.2.2
AZU MA , R. T., BAI LL OT, Y., BEHRINGER, R.,
FEINER, S. K., JULIER, S. AN D MAC INTYRE, B.
Recent advances in augmented reality. Computer
Graphics and Applications 21, 6 (Nov./Dec. 2001),
pp. 34–47. 1, 1.1, 1.2, 27, 4, 4
AZU MA , R. T., NEE LY III, H ., DAILY, M.
AND LEO NARD , J. Performance analysis of an out-
door augmented reality tracking system that relies
upon a few mobile beacons. In ISMAR’06: Proc. 5th
Int’l Symp. on Mixed and Augmented Reality (Santa
Barbara, CA, USA, Oct. 2006), pp. 101–104. 2.2.2,
3.2
BAH L, P. AND PADM ANAB H AN , V. N. RADAR: an in-
building RF-based user location and tracking system.
In Proc. IEEE Infocom (Tel Aviv, Israel, 2000), IEEE
CS Press, pp. 775–784. ISBN 0-7803-5880-5. 2.2.2
BAI LL OT, Y., BRO WN , D. A N D JULIER, S . Authoring of
physical models using mobile computers. In ISWC’01:
Proc. 5th Int’l Symp. on Wearable Computers (Wash-
ington, DC, USA, 2001), IEEE CS Press, p. 39. ISBN
0-7695-1318-2. 3.2
BARTIE, P. J. AND MAC K AN ES S, W. A. Development
of a speech-based augmented reality system to support
exploration of cityscape. Trans. in GIS 10, 1 (2006),
pp. 63–86. 3.1
BEHRINGER, R., TAM, C ., MCGE E, J ., SU N-
DAR ES WAR AN , S . AND VASSILIOU, M. Wearable
augmented reality testbed for navigation and control,
built solely with commercial-off-the-shelf (COTS)
hardware. In ISAR’00: Proc. Int’l Symp. Augmented
Reality (Washington, DC, USA, Oct. 2000), IEEE CS
Press, pp. 12–19. 2.4
BEL L, B ., F EINER, S. AND H¨
OL LE RE R , T. View
management for virtual and augmented reality.
In UIST’01: Proc. ACM Symp. on User Interface Soft-
ware and Technology (Orlando, FL, 2001), vol. 3 of
CHI Letters, ACM Press, pp. 101–110. 2.2.1
BENA LI-KH OU D JA, M ., H A FE Z, M ., ALEXANDRE, J.-
M. A ND KH ED DA R, A. Tactile interfaces: a state-of-
the-art survey. In ISR’04: Proc. 35th Int’l Symp. on
Robotics (Paris, France, Mar. 2004). 2.3
BEN FO RD , S., GREENHALGH, C. , REY NA RD , G. ,
BROWN, C. AND KO L EVA, B. Understanding and
constructing shared spaces with mixed-reality bound-
aries. ACM Trans. on Computer-Human Interaction 5,
3 (Sep. 1998), pp. 185–223. 3.5
BEN GL ER , K . A ND PAS SARO , R. Augmented reality
in cars: Requirements and constraints. In ISMAR’04:
Proc. 3rd Int’l Symp. on Mixed and Augmented Reality
(2004). 4
BILLINGHURST, M. Augmented reality in education.
New Horizons in Learning 9, 1 (Oct. 2003). 3.6
BILLINGHURST, M. AND KATO , H. Collaborative
mixed reality. In ISMR’99: Proc. Int’l Symp. on Mixed
Reality (Yokohama, Japan, Mar. 1999), Springer Ver-
lag, pp. 261–284. 3.5
BILLINGHURST, M., KATO , H. AN D POUPYREV, I. The
MagicBook–moving seamlessly between reality and
virtuality. Computer Graphics and Applications 21,
3 (May/June 2001), pp. 6–8. 3.5
BIMBER, O. A ND RASKAR, R. Modern approaches to
augmented reality. In SIGGRAPH’05 Tutorial on Spa-
tial AR (2005a). 3, 4, 5, 6, 8, 9
BIMBER, O. A ND RASKAR, R . Spatial Augmented Re-
ality: Merging Real and Virtual Worlds. A. K. Peters,
Ltd., Wellesley, MA, USA, 2005b. ISBN 1-56881-
230-2. 2.1.2
BIOCCA, F. A. AND ROL L AN D, J. P. Virtual eyes can re-
arrange your body–adaptation to visual displacement
in see-through, head-mounted displays. Presence 7, 3
(June 1998), pp. 262–277. 2.1.2, 4
BROL L, W., LI ND T, I. , OH L EN BU RG, J.,
WIT TK ¨
AMPER, M., YUAN , C. , NO VOT NY, T.,
FATAH GEN . SCHIECK, A ., MOT T RA M, C .
AND STROTH MA NN , A. ARTHUR: A collabo-
rative augmented environment for architectural design
and urban planning. Journal of Virtual Reality and
Broadcasting 1, 1 (Dec. 2004). 29, 3.5
BUC HM AN N , V., VIOLICH, S. , BILLINGHURST,
M. AND COCKBURN, A. FingARtips: Ges-
ture based direct manipulation in augmented reality.
In GRAPHITE’04: Proc. 2nd Int’l Conf. on Com-
puter graphics and interactive techniques (Singapore,
2004), ACM Press, pp. 212–221. ISBN 1-58113-883-
0. 2.3
19

References
BUT Z, A ., H ¨
OLLERER, T., FEINER, S ., MAC IN-
TY RE , B. AND BES HER S, C. Enveloping users
and computers in a collaborative 3D augmented real-
ity. In IWAR’99: Proc. 2nd Int’l Workshop on Aug-
mented Reality (Washington, DC, USA, 1999), IEEE
CS Press, pp. 35–44. ISBN 0-7695-0359-4. 3.5
CAK MA KC I , O. AND ROL LAN D, J. Head-worn dis-
plays: A review. Display Technology 2, 3 (Sep. 2006),
pp. 199–216. Invited. 2.1.3
CAS TRO , P., CHIU, P., KREM EN EK , T. A ND MU NT Z,
R. A probabilistic room location service for wire-
less networked environments. In UbiComp’01: Proc.
Ubiquitous Computing (New York, NY, USA, 2001),
vol. 2201 of Lecture Notes in Computer Science, ACM
Press, pp. 18–35. 2.2.2
CAUD EL L, T. P. AND MIZELL, D. W. Augmented
reality: An application of heads-up display technol-
ogy to manual manufacturing processes. In Proc.
Hawaii Int’l Conf. on Systems Sciences (Washington,
DC, USA, 1992), IEEE CS Press. 1.2
CAVAL LA RO, R. The foxtrax hockey puck tracking
system. Computer Graphics and Applications 17, 2
(Mar./Apr. 1997), pp. 6–12. 3.4
CHA NG , A. AND O’SULLIVAN, C . Audio-haptic feed-
back in mobile phones. In CHI’05: Proc. SIGCHI
Conf. on Human Factors in Computing Systems (Port-
land, OR, USA, 2005), ACM Press, pp. 1264–1267.
ISBN 1-59593-002-7. 2.1.1
CHE OK , A. D ., WAN, F. S., YANG, X., W EI H UA, W.,
HUAN G, L . M., BILLINGHURST, M. AND KATO, H.
Game-city: A ubiquitous large area multi-interface
mixed reality game space for wearable computers.
IEEE CS Press. ISBN 0-7695-1816-8/02. 3.4
CHIA, K. W., CH E OK , A. D. AND PRINCE, S. J. D.
Online 6 DOF augmented reality registration from nat-
ural features. In ISMAR’02: Proc. Int’l Symp. on
Mixed and Augmented Reality (Darmstadt, Germany,
Oct. 2002), IEEE CS Press. 2.2.2
COL LE TT, T. H. J. A ND MA CDO NA LD , B . A. De-
veloper oriented visualisation of a robot program: An
augmented reality approach. In HRI06 (Salt Lake City,
Utah, USA, Mar. 2006), ACM Press, pp. 49–56. ISBN
1-59593-294-1/06/0003. 23, 3.2
COM PO RT, A. I., M ARCHAND, E. AN D CHAUMETTE,
F. A real-time tracker for markerless augmented real-
ity. In ISMAR’03: Proc. 2nd Int’l Symp. on Mixed and
Augmented Reality (Tokyo, Japan, Oct. 2003), IEEE
CS Press. 2.2.2
CORREIA, N. A N D ROM ERO, L. Storing user experi-
ences in mixed reality using hypermedia. The Visual
Computer 22, 12 (2006), pp. 991–1001. 2.4
CRABTREE, A., BE NF O RD , S. , RODDEN, T., G R EE N-
HALGH, C., FLINTHAM, M ., A NA STA SI , R.,
DROZ D, A ., ADA M S, M., ROW-FARR, J., TANDA -
VANITJ , N. AND STE ED , A. Orchestrating a mixed
reality game ‘on the ground’. In Proc. CHI Conf. on
Human Factors in Computing Systems (Vienna, Aus-
tria, Apr. 2004), ACM Press. 3.4
CURTIS, D., MIZ EL L , D., GR UE NB AUM , P.
AND JAN IN , A. Several devils in the details:
Making an AR application work in the airplane fac-
tory. In IWAR’98: Proc. Int’l Workshop on Augmented
Reality (Natick, MA, USA, 1998), A. K. Peters, Ltd.,
pp. 47–60. ISBN 1-56881-098-9. 3.2
DIX , A. , FI N LAY, J. E., AB OW D, G . D. AND BEA LE ,
R. Human-Computer Interaction, 3rd ed. Pearson
Education Ltd., Harlow, England, 2004. ISBN 0130-
46109. ∗
DRASCIC, D. AND MILGRAM, P. Perceptual issues in
augmented reality. In Proc. SPIE, Stereoscopic Dis-
plays VII and Virtual Systems III (Bellingham, WA,
USA, 1996), vol. 2653, SPIE Press, pp. 123–134. 4
ELL IS , S. R ., B REANT, F., MANGES, B., JAC O BY, R.
AND ADE LS TE I N, B . D . Factors influencing opera-
tor interaction with virtual objects viewed via head-
mounted seethrough displays: Viewing conditions and
rendering latency. In VRAIS’97: Proc. Virtual Reality
Ann. Intl Symp. (Washington, DC, USA, 1997), IEEE
CS Press, pp. 138–145. ISBN 0-8186-7843-7. 4
FARRINGDON, J., M O OR E, A . J., TIL BU RY, N.,
CHURCH, J. AND BIEMOND, P. D. Wearable sen-
sor badge and sensor jacket for context awareness.
In ISWC’99: Proc. 3rd Int’l Symp. on Wearable Com-
puters (San Francisco, CA, 1999), IEEE CS Press,
pp. 107–113. 2.3
FEINER, S., MAC INTYRE, B. , H ¨
OL LE RE R , T.
AND WEB ST ER , A. A touring machine: Prototyp-
ing 3D mobile augmented reality systems for explor-
ing the urban environment. In ISWC’97: Proc. 1st
Int’l Symp. on Wearable Computers (Cambridge, MA,
1997), IEEE CS Press, pp. 74–81. 1.2
FEINER, S., MAC INTYRE, B. AN D H¨
OL LE RE R , T.
Mixed Reality: Merging Real and Virtual Worlds.
Ohmsha/Springer, Tokyo/New York, 1999, ch. Wear-
ing it out: First steps toward mobile augmented reality
systems, pp. 363–377. 3.2
20

References
FEINER, S. K . Augmented reality: A new way of seeing.
Scientific American (Apr. 2002). 1, 5
FERRARI, V., TUYTELAARS, T. AN D VAN GOO L , L.
Markerless augmented reality with a real-time affine
region tracker. In ISAR’01: Proc. 2nd Int’l Symp. Aug-
mented Reality (Washington, DC, USA, 2001), IEEE
CS Press. 2.2.2
FIAMBOLIS, P. Virtual retinal display (VRD) tech-
nology. Available at http://www.cs.nps.navy.
mil/people/faculty/capps/4473/projects/
fiambolis/vrd/vrd_full.html. Aug. 1999. 2.1.2
FIORENTINO, M., DE AMICIS, R ., M O NN O, G .
AND STORK , A . Spacedesign: A mixed reality
workspace for aesthetic industrial design. In IS-
MAR’02: Proc. Int’l Symp. on Mixed and Augmented
Reality (Darmstadt, Germany, Oct. 2002), IEEE CS
Press, p. 86. ISBN 0-7695-1781-1. 22, 3.2
FLINTHAM, M., ANAS TAS I, R., B ENF OR D, S ., HE M-
MINGS, T., CRABTREE, A ., GREENHALGH, C.,
RODDEN, T., TAN DAVAN IT J , N. , AD AM S, M .
AND ROW-FA RR , J. Where on-line meets on-the-
streets: Experiences with mobile mixed reality games.
2003. 3.4
FOXL IN , E. AND HARRINGTON, M. Weartrack: A self-
referenced head and hand tracker for wearable com-
puters and portable VR. In ISWC’00: Proc. 4th Int’l
Symp. on Wearable Computers (Atlanta, GA, 2000),
IEEE CS Press, pp. 155–162. 2.3
FRIEDRICH, W. AND WOHLGEMUTH, W. ARVIKA–
augmented reality for development, production and
servicing. ARVR Beitrag, 2002. 3.2, 3.2, 3.2
FUC HS , H. , LIVINGSTON, M. A., RASKAR, R.,
COLUCCI, D., KE LL E R, K., S TATE, A ., CR AW-
FO RD , J. R. , R ADEMACHER, P., DR AK E, S. H.
AND MEY ER , A . A. Augmented reality visualization
for laparoscopic surgery. In MICCAI’98: Proc. 1st
Intl Conf. Medical Image Computing and Computer-
Assisted Intervention (Heidelberg, Germany, 1998),
Springer-Verlag, pp. 934–943. ISBN 3-540-65136-5.
25, 3.3
G. HEIDEMANN, H. B EK E L, I . B. AND RI T TE R, H . In-
teractive online learning. Pattern Recognition and Im-
age Analysis 15, 1 (2005), pp. 55–58. 3.6
GET TI NG , I. The global positioning system. Spectrum
30, 12 (1993), pp. 36–47. 2.2.2
GOEBBELS, G., TRO C HE , K. , BRAU N, M ., IVAN OVI C,
A., GRAB, A ., V ON L¨
OB TOW, K., ZEI LHO FE R,
H. F., S ADE R, R ., THIERINGER, F., A LBRECHT, K.,
PRAXMARER, K., KE EV E , E., HA N SS EN , N., KROL,
Z. A ND HASENBRINK, F. Development of an aug-
mented reality system for intra-operative navigation in
maxillo-facial surgery. In Proc. BMBF Statustagung
(Stuttgart, Germany, 2001), pp. 237–246. 2.1.3
GORDON, I. A ND LO WE , D. G. What and where: 3D
object recognition with accurate pose. In ISMAR’04:
Proc. 3rd Int’l Symp. on Mixed and Augmented Real-
ity (Arlington, VA, USA, Nov. 2004), IEEE CS Press.
2.2.2
HAS VOL D , P. In-the-field health informatics. In The
Open Group conf. (Paris, France, 2002). 3.3
HAYWARD , V., AS TL EY, O. , CR UZ -H E RNA N DE Z, M .,
GRA NT, D . AND ROB L ES -D ELATORR E, G . Hap-
tic interfaces and devices. Sensor Review 24 (2004),
pp. 16–29. 2.3
HEN RYS SO N, A., BILLINGHURST, M. AND OLL IL A,
M. Face to face collaborative AR on mobile phones.
In ISMAR’05: Proc. 4th Int’l Symp. on Mixed and Aug-
mented Reality (Vienna, Austria, Oct. 2005), IEEE CS
Press, pp. 80–89. ISBN 0-7695-2459-1. 28, 3.4
H¨
OL LE RE R , T., FEINER, S. AND PAVLI K , J. Situated
documentaries: Embedding multimedia presentations
in the real world. In ISWC’99: Proc. 3rd Int’l Symp.
on Wearable Computers (San Francisco, CA, 1999),
IEEE CS Press, pp. 79–86. 3.1
H¨
OL LE RE R , T. H. AND FEINER, S. K. Telegeoinformat-
ics: Location-Based Computing and Services. Taylor
& Francis Books Ltd., 2004, ch. Mobile Augmented
Reality. 2.1.3, 2.2.2, 2.2.2, 2.3, 2.3, 2.4, 3.1, 4
HUGHES, C. E ., STA PL ET ON , C. B. , HUGHES, D. E.
AND SMITH, E. M . Mixed reality in education, enter-
tainment, and training. Computer Graphics and Appli-
cations (Nov./Dec. 2005), pp. 24–30. 3.6
HUGHES, C. E., S TAP LE TO N , C. B. AN D O’CONNOR,
M. The evolution of a framework for mixed reality
experiences. In Emerging Technologies of Augmented
Reality: Interfaces and Design (Hershey, PA, USA,
Nov. 2006), Idea Group Publishing. 2.1.1
ISH II , H . AND ULLMER, B. Tangible bits: Towards
seamless interfaces between people, bits and atoms.
In CHI’97: Proc. SIGCHI Conf. on Human Factors in
Computing Systems (New York, NY, USA, Mar. 1997),
ACM Press, pp. 234–241. 2.3
JACO B, R. J. K . What is the next generation of
human-computer interaction? In CHI’06: Proc.
21

References
SIGCHI Conf. on Human Factors in Computing Sys-
tems (Montr´
eal, Qu´
ebec, Canada, Apr. 2006), ACM
Press, pp. 1707–1710. ISBN 1-59593-298-4. 1
JEBA RA, T., E Y ST ER , C., WEAVE R, J ., STAR N ER ,
T. AND PENTLAND, A. Stochasticks: Augment-
ing the billiards experience with probabilistic vision
and wearable computers. In ISWC’97: Proc. 1st
Int’l Symp. on Wearable Computers (Washington, DC,
USA, 1997), IEEE CS Press, pp. 138–145. 3.4
JULIER, S. A ND BISHOP, G., E.Computer Graphics and
Applications, Special Issue on Tracking, vol. 6. IEEE
CS Press, Washington, DC, USA, 2002. 2.2.2
JULIER, S., KING, R ., C O LB ERT, B., D URBIN, J.
AND ROS EN BL U M, L. The software architecture of a
real-time battlefield visualization virtual environment.
In VR’99: Proc. IEEE Virtual Reality (Washington,
DC, USA, 1999), IEEE CS Press, pp. 29–36. ISBN
0-7695-0093-5. 3.2
JULIER, S. J., LAN ZA GO RTA, M., BAI LL OT, Y.,
ROSENBLUM, L. J., FEINER, S ., H ¨
OL LE RE R , T.
AND SES TI TO , S. Information filtering for mobile
augmented reality. In ISAR’00: Proc. Int’l Symp. Aug-
mented Reality (Washington, DC, USA, 2000), IEEE
CS Press, pp. 3–11. 3.2
KAP LA N- LEISERSON, E. Trend: Augmented reality
check. Learning Circuits (Dec. 2004). 3.2, 3.6
KAS AI , I. , TAN IJ I RI , Y., ENDO, T. A ND UE DA, H.
A forgettable near eye display. In ISWC’00: Proc.
4th Int’l Symp. on Wearable Computers (Atlanta, GA,
USA, 2000), IEEE CS Press, pp. 115–118. 2.1.3
KAUFMANN, H. Construct3D: an augmented reality
application for mathematics and geometry education.
In MULTIMEDIA’02: Proc. 10th ACM Int’l Conf.
on Multimedia (Juan-les-Pins, France, 2002), ACM
Press, pp. 656–657. ISBN 1-58113-620-X. 3.6
KAUFMANN, H., SCH MAL ST IE G , D. AN D WAGNER ,
M. Construct3D: A virtual reality application for
mathematics and geometry education. Education and
Information Technologies 5, 4 (Dec. 2000), pp. 263–
276. 30, 3.6
KIM , S. , KI M , H., EO M , S., MAHALIK, N. P.
AND AHN , B . A reliable new 2-stage distributed inter-
active tgs system based on gis database and augmented
reality. In IEICE Trans. on Information and Systems,
vol. E89-D. Jan. 2006. 2.2.1, 3.1
KIYO KAWA, K., BILLINGHURST, M., C AMPBELL, B.
AND WOO DS , E. An occlusion-capable optical see-
through head mount display for supporting co-located
collaboration. In ISMAR’03: Proc. 2nd Int’l Symp.
on Mixed and Augmented Reality (Tokyo, Japan, Oct.
2003), IEEE CS Press, p. 133. ISBN 0-7695-2006-5.
2.1.2
KLI NK ER , G., STRICKER, D. AND REINERS, D. Aug-
mented Reality and Wearable Computers. Lawrence
Erlbaum Press, 2001, ch. Augmented Reality for Ex-
terior Construction Applications. 3.2
KOC H, R ., KOE SE R , K., STRECKEL, B. AND EVE RS -
SEN NE , J. -F. Markerless image-based 3D tracking for
real-time augmented reality applications. 2005. 2.2.2
KOMMERS, P. A. AN D ZHIMING, Z. Virtual reality
for education. Available at http://projects.edte.
utwente.nl/proo/kommers.htm. 3.6
KRITZENBERGER, H., WINKLER, T. AND HERC ZE G,
M. Collaborative and constructive learning of ele-
mentary school children in experiental learning spaces
along the virtuality continuum. In Mensch & Com-
puter 2002: Vom interaktiven Werkzeug zu koopera-
tiven Arbeits- und Lernwelten, M. Herczeg, W. Prinz,
and H. Oberquelle, Eds. B. G. Teubner, Stuttgart, Ger-
many, 2002. 3.6
KUMAR, M. AN D WINOGRAD, T. GUIDe: Gaze-
enhanced UI design. In CHI’07: Proc. SIGCHI Conf.
on Human Factors in Computing Systems (San Jose,
CA, Apr./May 2007). 2.3
LIAROKAPIS, F., MOUR KO US SI S, N., WH IT E , M.,
DAR CY, J., SI FN I OTI S , M., PETRIDIS, P., BA S U, A.
AND LIS TE R, P. F. Web3D and augmented reality to
support engineering education. World Trans. on Engi-
neering and Technology Education 3, 1 (2004), pp. 1–
4. UICEE. 30, 3.6
LICKLIDER, J. Man-computer symbiosis. IRE Trans. on
Human Factors in Electronics 1 (1960), pp. 4–11. 2.3,
5
LOO MI S, J ., GOLLEDGE, R . AN D KL ATZK Y, R. Per-
sonal guidance system for the visually impaired using
GPS, GIS, and VR technologies. In Proc. Conf. on Vir-
tual Reality and Persons with Disabilities (Millbrae,
CA, 1993). 1.2
MAGN EN AT-THALMANN, N. AN D PAPAG IA NN AK IS,
G. Virtual worlds and augmented reality in cultural
heritage applications. 2005. 3.1
22

References
MANN, S. Wearable computing: A first step toward per-
sonal imaging. Computer 30, 2 (Feb. 1997), pp. 25–32.
1.2, 2.4
MATYS CZ OK, C., R ADKO WS KI , R. AND BE R SS EN -
BRU EG G E, J. AR-bowling: immersive and realistic
game play in real environments using augmented real-
ity. In ACE’04: Proc. ACM SIGCHI Int’l Conf. on Ad-
vances in Computer Entertainment technology (Singa-
pore, 2004), ACM Press, pp. 269–276. ISBN 1-58113-
882-2. 3.4
MAYOL, W. W. AN D MUR RAY, D. W. Wearable
hand activity recognition for event summarization.
In ISWC’05: Proc. 9th Int’l Symp. on Wearable Com-
puters (Osaka, Japan, Oct. 2005), IEEE CS Press. 2.3
MERT EN , M. Erweiterte realit ¨
at–verschmelzung zweier
welten. Deutsches ¨
Arzteblatt 104, 13 (Mar. 2007). 26,
3.3
MILGRAM, P. AND KISHINO, F. A taxonomy of mixed
reality visual displays. IEICE Trans. on Information
and Systems E77-D, 12 (Dec. 1994), pp. 13–21. 1.1
MIZELL, D. Fundamentals of Wearable Computers and
Augumented Reality. Lawrence Erlbaum Assoc, Mah-
wah, NJ, 2001, ch. Boeing’s wire bundle assembly
project, pp. 447–467. 3.2
M¨
OHRING, M., LESSIG, C . AND BIMBER, O. Video
see-through AR on consumer cell-phones. In IS-
MAR’04: Proc. 3rd Int’l Symp. on Mixed and Aug-
mented Reality (Arlington, VA, USA, Nov. 2004),
IEEE CS Press, pp. 252–253. ISBN 0-7695-2191-6.
2.1.3
NAIMARK, L. AND FOX LI N, E . Circular data ma-
trix fiducial system and robust image processing for
a wearable vision-inertial self-tracker. In ISMAR’02:
Proc. Int’l Symp. on Mixed and Augmented Reality
(Darmstadt, Germany, Oct. 2002), IEEE CS Press.
2.2.2
NAR ZT, W., POMBERGER, G. , FER SC HA , A., KOL B,
D. , M ¨
UL LE R, R ., WIEGHARDT, J ., H ¨
ORT NE R, H .
AND LINDINGER, C . Pervasive information acquisi-
tion for mobile ar-navigation systems. In WMCSA’03:
Proc. 5th Workshop on Mobile Computing Systems and
Applications (Oct. 2003), IEEE CS Press, pp. 13–20.
ISBN 0-7695-1995-4/03. 21, 3.1
NAR ZT, W., POMBERGER, G. , FER SC HA , A., KOL B,
D. , M ¨
UL LE R, R ., WIEGHARDT, J ., H ¨
ORT NE R, H .
AND LINDINGER, C . Augmented reality navigation
systems. Universal Access in the Information Society
4, 3 (2006), pp. 177–187. 2.1.3, 3.1
NAVAB , N., BA NI -H ASH EM I, A. AND MITSCHKE, M.
Merging visible and invisible: Two camera-augmented
mobile C-arm (CAMC) applications. In IWAR’99:
Proc. 2nd Int’l Workshop on Augmented Reality
(Washington, DC, USA, 1999), IEEE CS Press,
pp. 134–141. ISBN 0-7695-0359-4. 3.3
OGI , T., YAMADA, T., YAMAMOTO, K. A ND HI ROSE ,
M. Invisible interface for immer sive virtual world.
In IPT’01: Proc. Immersive Projection Technology
Workshop (Stuttgart, Germany, 2001), pp. 237–246.
2.1.3
OHSHIMA, T., SATO H, K., YAMAMOTO, H.
AND TAMURA, H. AR2Hockey: A case study
of collaborative augmented reality. In VRAIS’98:
Proc. Virtual Reality Ann. Intl Symp. (Washington,
DC, USA, 1998), IEEE CS Press, pp. 268–275. 3.4
OHSHIMA, T., SATO H, K., YAMAMOTO, H.
AND TAMURA, H . RV-Border Guards: A multi-
player mixed reality entertainment. Trans. Virtual
Reality Soc. Japan 4, 4 (1999), pp. 699–705. 3.4
OLWAL, A . AN D FEINER, S . Unit: modular develop-
ment of distributed interaction techniques for highly
interactive user interfaces. In GRAPHITE’04: Proc.
2nd Int’l Conf. on Computer graphics and interactive
techniques (Singapore, 2004), ACM Press, pp. 131–
138. ISBN 1-58113-883-0. 3.2
PAN, Z ., Z HIGENG, A. D., YANG , H., ZHU , J.
AND SHI , J . Virtual reality and mixed reality for vir-
tual learning environments. Computers & Graphics
30, 1 (Feb. 2006), pp. 20–28. 3.6
PAPAGIANNAKIS, G., SCHERTENLEIB, S.,
OKEN NE DY, B., ARE VALO -P OIZ AT, M.,
MAGN EN AT-THALMANN, N., STO DD ART, A.
AND THALMANN, D. Mixing virtual and real scenes
in the site of ancient pompeii. Computer Animation
and Virtual Worlds 16 (2005), pp. 11–24. 3.1
PATEL, S . N., RE KI MO TO , J . AND AB OWD, G. D .
iCam: Precise at-a-distance interaction in the physical
environment. Feb. 2006. 2.2.1
PICARD, R. W. Affective Computing. MIT Press, Cam-
bridge, MA, USA, 1997. 2.3
PIEKARSKI, W. AND THOM AS , B. Tinmith-Metro: New
outdoor techniques for creating city models with an
augmented reality wearable computer. In ISWC’01:
Proc. 5th Int’l Symp. on Wearable Computers (Z¨
urich,
Switzerland, 2001), IEEE CS Press, pp. 31–38. 2.2.1,
3.4
23

References
PIEKARSKI, W. AND THOM AS , B. ARQuake: the out-
door augmented reality gaming system. Communica-
tions of the ACM 45, 1 (2002), pp. 36–38. 3.4
PIEKARSKI, W., GU NT HER , B. A N D THO MA S, B. In-
tegrating virtual and augmented realities in an outdoor
application. In IWAR’99: Proc. 2nd Int’l Workshop
on Augmented Reality (Washington, DC, USA, 1999),
IEEE CS Press, pp. 45–54. 3.2, 3.4
PIETRZAK, P., ARYA, M., JOS EP H , J. V. AN D PATEL,
H. R. H. Three-dimensional visualization in laparo-
scopic surgery. British Journal of Urology Interna-
tional 98, 2 (Aug. 2006), pp. 253–256. 3.3
POUPYREV, I., TAN, D ., BILLINGHURST, M., KATO ,
H., REGENBRECHT, H. AND TE TS UTANI , N. Tiles:
A mixed reality authoring interface. 2001. 2.3
RAA B, F. H., BL OO D, E. B. , ST EI NER , T. O.
AND JON ES , H . R . Magnetic position and orientation
tracking system. Trans. on Aerospace and Electronic
Systems 15, 5 (1979), pp. 709–717. 2.2.2
RASKAR, R., WE LC H , G. AND CH EN , W.-C. Table-top
spatially-augmented realty: Bringing physical models
to life with projected imagery. In IWAR’99: Proc. 2nd
Int’l Workshop on Augmented Reality (Washington,
DC, USA, 1999), IEEE CS Press, pp. 64–70. ISBN
0-7695-0359-4. 2.1.2
RASKAR, R., VAN BAAR, J., BE ARD SL EY, P.,
WIL LWACHE R, T., RAO, S. AND FORLINES, C. iL-
amps: geometrically aware and self-configuring pro-
jectors. ACM Trans. on Graphics 22, 3 (2003),
pp. 809–818. 2.1.3
REK IM OTO , J. Transvision: A hand-held augmented re-
ality system for collaborative design. In VSMM’96:
Proc. Virtual Systems and Multimedia (Gifu, Japan,
1996), Intl Soc. Virtual Systems and Multimedia,
pp. 85–90. 3.5
REK IM OTO , J. NaviCam: A magnifying glass approach
to augmented reality. Presence 6, 4 (Aug. 1997),
pp. 399–412. 3.1
REK IM OTO , J. AND SAITOH, M . Augmented surfaces:
A spatially continuous workspace for hybrid comput-
ing environments. In CHI’99: Proc. SIGCHI Conf.
on Human Factors in Computing Systems (Pittsburgh,
Pennsylvania, USA, 1999), ACM Press, pp. 378–385.
ISBN 0-201-48559-1. 3.5
ROL LA ND , J. P. AND FUCH S, H. Optical versus video
seethrough head-mounted displays in medical visual-
ization. Presence 9, 3 (June 2000), pp. 287–309. 4
ROL LA ND , J. P., B IOCCA, F., HAMZA-L U P, F., HA,
Y. AND MARTIN S, R. Development of head-mounted
projection displays for distributed, collaborative, aug-
mented reality applications. Presence 14, 5 (2005),
pp. 528–549. 2.1.3
SANDOR, C. AND KLI NK E R, G. A rapid prototyp-
ing software infrastructure for user interfaces in ubiq-
uitous augmented reality. Personal and Ubiquitous
Computing 9, 3 (May 2005), pp. 169–185. 3.2
SCHMALSTIEG, D., FU HR M AN N, A . A ND HE SI NA , G.
Bridging multiple user interface dimensions with aug-
mented reality. In ISAR’00: Proc. Int’l Symp. Aug-
mented Reality (2000). 2.2.2, 15, 2.3
SCHOWENGERDT, B. T., S EIBEL, E. J., KE LLY, J. P.,
SILVERMAN, N. L. AN D FURN ES S II I, T. A. Binoc-
ular retinal scanning laser display with integrated fo-
cus cues for ocular accommodation. In Proc. SPIE,
Electronic Imaging Science and Technology, Stereo-
scopic Displays and Applications XIV (Bellingham,
WA, USA, Jan. 2003), vol. 5006, SPIE Press. 2.1.2,
7
STARN ER, T., M A NN , S. , RHODES, B., LEVINE, J .,
HEA LE Y, J., KIRSCH, D. , PICARD, R., AN D PENT-
LA ND , A. Augmented reality through wearable com-
puting. Presence 6, 4 (1997), pp. 386–398. 1.2, 3.1
STE TT EN , G., CHIB, V., HILDEBRAND, D.
AND BURSEE, J . Real time tomographic reflection:
Phantoms for calibration and biopsy. In ISAR’01:
Proc. 2nd Int’l Symp. Augmented Reality (Washington,
DC, USA, 2001), IEEE CS Press, pp. 11–19. ISBN
0-7695-1375-1. 2.1.3
STOA KL EY, R., CON WAY, M. AND PAUS C H, R. Vir-
tual reality on a WIM: Interactive worlds in miniature.
In CHI’95: Proc. SIGCHI Conf. on Human Factors in
Computing Systems (1995), pp. 265–272. 2.2.1
SUGIHARA, T. AND MIYASATO, T. A lightweight 3-D
HMD with accommodative compensation. In SID’98:
Proc. 29th Society for Information Display (San Jose,
CA, USA, May 1998), pp. 927–930. 2.1.2
SUTHERLAND, I. E . A head-mounted three-dimensional
display. In Proc. Fall Joint Computer Conf. (Washing-
ton, DC, 1968), Thompson Books, pp. 757–764. 2,
1.2, 2.2.2
SZA LAV ´
ARI, Z., SCHMALSTIEG, D ., F U HR MA NN, A.
AND GERVAUTZ , M. Studierstube: An environment
for collaboration in augmented reality. Virtual Reality
3, 1 (1998), pp. 37–49. 2.4, 3.5
24

References
TAKAG I , A. , YAMAZAKI, S., S AITO, Y.
AND TANIGUCHI, N. Development of a stereo
video see-through HMD for AR systems. In ISAR’00:
Proc. Int’l Symp. Augmented Reality (M¨
unchen,
Germany, 2000), ACM Press. ISBN 0-7695-0846-4.
2.1.2
TAMURA, H. Steady steps and giant leap toward practi-
cal mixed reality systems and applications. In VAR’02:
Proc. Int’l Status Conf. on Virtual and Augmented Re-
ality (Leipzig, Germany, Nov. 2002). 22, 3.2, 3.4, 29,
3.5
TAMURA, H., YAMAMOTO, H. A ND KATAYAMA, A .
Mixed reality: Future dreams seen at the border be-
tween real and virtual worlds. Computer Graphics and
Applications 21, 6 (Nov./Dec. 2001), pp. 64–70. 2.1.3,
22, 3.2, 3.4
TANG , A. , OWEN, C ., B IOCCA, F. AND MOU , W.
Comparative effectiveness of augmented reality in ob-
ject assembly. In CHI’03: Proc. SIGCHI Conf. on Hu-
man Factors in Computing Systems (Ft. Lauderdale,
Florida, USA, 2003), ACM Press, pp. 73–80. ISBN
1-58113-630-7. 4
THO MA S, B ., CL OS E, B., DONOGHUE, J ., SQ UI RE S ,
J., DE BO ND I, P., M ORRIS, M. AND PIEKARSKI, W.
ARQuake: An outdoor/indoor augmented reality first
person application. In ISWC’00: Proc. 4th Int’l Symp.
on Wearable Computers (2000), pp. 139–146. ISBN
0-7695-0795-6. 3.4
T¨
ONNIS, M., SANDOR, C ., K L IN KE R, G., LA NGE , C.
AND BUBB, H . Experimental evaluation of an aug-
mented reality visualization for directing a car driver’s
attention. In ISMAR’05: Proc. 4th Int’l Symp. on
Mixed and Augmented Reality (Vienna, Austria, Oct.
2005), IEEE CS Press, pp. 56–59. ISBN 0-7695-2459-
1. 21, 3.1
VAISSIE, L. A ND RO LL AND , J . Accuracy of ren-
dered depth in head-mounted displays: Choice of eye-
point locations. In Proc. AeroSense (Bellingham, WA,
2000), vol. 4021, SPIE, pp. 343–353. 4
VLAHAKIS, V., KARIGIANNIS, J. , TSOT RO S, M.,
GOUNARIS, M., A LME IDA, L., S TRICKER, D. ,
GLE UE , T., CHRISTOU, I. T., CARLUCCI, R.
AND IOANNIDIS, N. Archeoguide: first results of an
augmented reality, mobile computing system in cul-
tural heritage sites. In VAST’01: Proc. Conf. on Vir-
tual reality, archeology, and cultural heritage (Gly-
fada, Greece, 2001), ACM Press, pp. 131–140. ISBN
1-58113-447-9. 3.1
VOG T, S., KHAMENE, A . AN D SAU ER , F. Reality aug-
mentation for medical procedures: System architec-
ture, single camera marker tracking, and system eval-
uation. Int’l Journal of Computer Vision 70, 2 (2006),
pp. 179–190. 3.3
WAGNE R, D. A ND SCHMALSTIEG, D. First steps to-
wards handheld augmented reality. In ISWC’03: Proc.
7th Int’l Symp. on Wearable Computers (Washington,
DC, USA, 2003), IEEE CS Press, p. 127. ISBN 0-
7695-2034-0. 2.1.3
WEISER, M. The computer for the 21st century. Scien-
tific American 265, 3 (Sep. 1991), pp. 94–104. 5
WIL LE RS , D. Augmented reality at airbus. In IS-
MAR’06: Proc. 5th Int’l Symp. on Mixed and Aug-
mented Reality (Santa Barbara, CA, USA, Oct. 2006),
Airbus CIMPA GmbH, IEEE CS Press. 2.2.2, 24, 3.2,
5
WINKLER, T., KRITZENBERGER, H. AND HE RC ZE G,
M. Mixed reality environments as collaborative and
constructive learning spaces for elementary school
children. 2003. 3.6
25