Content uploaded by Andrzej Zarzycki
Author content
All content in this area was uploaded by Andrzej Zarzycki on Jan 26, 2015
Content may be subject to copyright.
Teaching and Designing for Augmented Reality
Andrzej Zarzycki
New Jersey Institute of Technology
www.njit.edu
andrzej.zarzycki@njit.edu
This paper discusses ways emerging interactive technologies are adopted by
designers and extended into areas of design, education, entertainment, and
commerce. It looks, in detail, at various project development stages and
methodologies used to engage design focused students into, often complex,
technological issues. The discussion is contextualized through a number of case
studies of mobile and marker-based augmented reality (AR) applications
developed by students. These applications include an app for a fashion based
social event that allows participants to preview recent collection additions, an
info-navigational app for the High Line elevated urban park in New York City, a
marker-based maze game, and an interior decorating interface to visualize
various furnishing scenarios. While a number of case studies will be discussed
from a developer perspective, the primary focus is on the concept and content
development, interface design, and user participation.
Keywords: Augmented Reality, AR, Gamification, Mobile Culture
INTRODUCTION
Let's pretend you're on your way to Manhattan to buy
some new clothes. Maybe you're looking to impress
someone on a date. Maybe you need an outfit to ace
that interview. Maybe you're looking to change your
style and try something new. Whatever the case may
be, you know New York fashion will not disappoint.
You arrive on 54th Street on Fifth Avenue early in
the afternoon. There's a ton of different stores in that
five-block radius. There's high-end fashion retail and
some typical name-brand stores. Some stores aren't in
your price range, but you might be interested in what
they have to offer. You're not sure where to shop first.
Decisions, decisions...Luckily, there's an app for that!
(students: Philip Caleja, Nick Haby, & Daniel Schittone)
The above excerpt from one of the scenarios for-
mulated by students as part of an augmented reality
(AR) project proposal explored ways mobile AR ap-
plications can enhance the fashion retail experience
by taking advantage of location-aware services and
electronic social networks. These applications also
address other, less tangible needs, such as the sense
of social happening and the excitement associated
with emerging technologies.
This paper looks into ways that emerging inter-
active technologies are being adopted by designers
and extended into the areas of tourism, education,
entertainment, and commerce. It discusses in de-
tail the project development stages and methodolo-
gies used to engage design-focused students into
often-complex technological issues. The discussion
is contextualized through a number of case studies
CAAD Education - Volume 1 - eCAADe 32 |357
of mobile and marker-based AR applications devel-
oped by students. These applications include an app
for fashion-based social events that allows partici-
pants to preview recent collection additions, an info-
navigational app for the High Line elevated urban
park in New York City, a marker-based maze game,
and an interior decorating interface to visualize vari-
ous furnishing scenarios.
While a number of case studies are discussed
from a developer perspective touching on technical
intricacies, the primary focus is on content develop-
ment, interface design, and user interaction consid-
erations. The paper also discusses pedagogy, con-
cept formation, and broader social and spatial narra-
tives.
AUGMENTED REALITY ENVIRONMENTS
Traditionally, AR environments employed two dis-
tinct types of data overlay. Marker-based environ-
ments employed distinct markers and, more recently,
images (image targets) to locate virtual data within
the physical world. A second approach associated
with mobile AR involved GPS, digital compass, and
accelerometer sensors to position users and the vir-
tual content around them.
AR-based applications provide an opportunity
reconnect and better realign virtual and physical
worlds through location awareness, enhanced data
overlays, and user-focused content. Unlike more
static forms of digital media, augmented reality, with
its interactive and context-aware functionalities, en-
gages users in more direct and meaningful ways. This
is evident not only in academia but also, and perhaps
primarily, in commerce and advertising. The interac-
tive print approach popularized by Layar or AR Lego
models [1] , associated with mainstream toy products
are successful because they extend the level of con-
sumer engagement. They provide additional infor-
mation and enticement (enticement) for consumers.
The same lessons of consumer engagement are di-
rectly applicable to design education and design ser-
vices.
While AR technology is routinely employed in
the form of data overlays providing supplementary
information for physical objects that are visible with
the unaided eye, it is also increasingly used to vi-
sualize less tangible structures and concepts such
as historical events, cultural phenomena, and scien-
tific processes. This can be seen in a number of re-
search projects and mobile apps that help users to
learn history and facilitate explorations of urban envi-
ronments. (Niedmermair, Ferschin, 2011) TimeWarp
(Herbst, et al, 2008) a mobile edutainment applica-
tion designed as an AR game situated in Cologne,
Germany, focuses on virtual reconstruction of historic
buildings by superimposing virtual imagery over cur-
rently existing structures. The application not only
shows no-longer-existing buildings as they originally
appeared but also visualizes design changes that oc-
curred over time to still-present structures. Along the
same lines, the Immersive Experience of Cultural Her-
itage project (Kim et al, 2009) uses an AR tour ap-
proach to provide tourists with a more realistic expe-
rience by placing virtual characters within historical
structures. Visitors to the heritage sites of Sajeong-
jeon and Gangnyeongjeon in Korea can use their mo-
bile devices to access additional facts associated with
the showcased physical content. While a similar ap-
proach is routinely used by many museums, this par-
ticular project does not rely on AR markers such as QR
codes. It implements visual camera tracking of the
rectangular display space to position its virtual actors
without a need for visually intrusive markers.
Figure 1
Augmented Reality
(AR) environment
as social and design
activism and urban
games.
358 |eCAADe 32 - CAAD Education - Volume 1
Virtual environment allow for explorations of in-
accessible or not-yet-materialized designs. They can
be precursors of future physical urban spaces and po-
tent drives in their realization. This is the case with AR
environments (fig.1) developed by Tremont Under-
ground Theater Space (TUTS) initiative [2]. This initia-
tive is using AR gaming media not only to popularize
ideas of the adaptive reuse of the abandoned pub-
lic infrastructure but also to build social constituency
and connect with general public (fig.2) The shifting
focus from virtual-reality (VR) environments toward
mixed-reality and AR frameworks indicates the reex-
amination of earlier visions of separated physical and
digital worlds. The emerging picture fuses both di-
mensions into a single continuum.
Figure 2
Mystery Spaces, a
map with POIs
arranged in the
form of the game
play.
The newfound physical context adopted by AR
games encourages players to push the boundaries of
social conventions and accepted public behavior.
Unlike more passive forms of entertainment such
as reality TV or even active-yet-confined console-
based games, the AR framework incorporates phys-
ical activities and social interaction as well as en-
couraging exploration, learning, and discovery. Fur-
thermore, as activities integrate digital media culture
within the built environment-cities-these games pro-
vide an insight into our physical-digital selves and
better understanding of ourselves and our commu-
nities.
CASE STUDIES
FashNYC
The FashNYC application helps its users make smarter
shopping choices while informing them about the
current fashion scene. Through the app, users have
access to online videos from each store's current col-
lection, watch interviews with designers, and access
garment information including sizes, colors, materi-
als, prices, and availability. The app is geared to-
ward those interested primarily in high-end fashion
designers. FashNYC brings awareness to the fashion
industry, connects with seasonal events, creates an
exciting new shopping experience, and establishes
a presence in today's heightened mobile application
culture.
The app was developed for the Layar AR browser
using PHP programming and MySQL as a database.
The student team focused not only on creation of in-
dividual assets and associated Web pages (fig.3) but
also on the overall packaging, user experience (fig.4),
and final layout for the navigational map (fig.5). Since
the seamless connectivity to various social networks
and fashion-based websites was key here, the app
user interface became a critical part of the overall ef-
fort.
Figure 3
Virtual changing
room with FashNYC
CAAD Education - Volume 1 - eCAADe 32 |359
While the student team developed a fully func-
tional prototype [3], they also proposed the next level
of functionality that went beyond the scope of the
class and student technical competencies.
Figure 4
Mobile interface for
FashNYC
Figure 5
App navigation
map
This was an important part of the overall design
strategy, where each team was asked to develop the
project concept initially and later rework it into a
more formalized business proposal with future devel-
opment stages indicated. One of the proposed fea-
tures that could be implemented with Layar's interac-
tive graphics was garment (image) recognition. The
proposed app functionality was expressed in the fol-
lowing statement.
Let's say you're looking for a new blue suit. You see
one you like in the window of Versace. To be quite frank,
you can't afford it, but you really like the color and the
style. You can take a picture of the suit and have the ap-
plication search for it. As the application searches for
it, you take the time to watch a video of the Fall/Winter
2013 runway show where the suit wasfirst featured. You
notice you like the way it looks when on the runway, but
the blue one that was shown looks even better. Once the
suit has been found in the search, you can try to find sim-
ilar garments in nearby stores. You're in luck! That ap-
plication located a similar blue tight-fitting suit in Zara
across the street for half the price! Bam! Success. (stu-
dents: Philip Caleja, Nick Haby, & Daniel Schittone)
As with any new app or a product relying on so-
cial interconnectivity and input, the key issue is to de-
velop a critical mass of active users who would prop-
agate its virtual life. This is a major challenge facing
many new media products including an AR commu-
nity like the one proposed by students. The strategy
to address this impediment and help with the future
commercialization of the app was to tie it to a partic-
ular event that is highly localized with a defined time
frame. The student team proposed to connect it to
the Fashion's Night Out (FNO) event or the New York
Fashion Week. While this was not implemented yet, it
provides a feasible strategy for launching an AR app
product that is highly contextualized with its theme,
location, and timing. This is also an approach used by
other mobile AR games such as Comfort of Strangers
[4] that rely on a critical mass of participants for their
success.
Highline Tour
Highline Tour [5] is a navigational and informational
mobile AR app geared toward visitors to the High
Line, an urban park in New York City (fig.6).
It provides users with historical and current in-
formation as well as plans for future developments.
Its location-aware functionality allows for sorting and
positioning data in relationship to the urban context.
It shows year-around activities with imagery of var-
ious plants and foliage reflecting seasonal changes
occurring in the park. Users of the app can look at
a particular section of a project and freely navigate
360 |eCAADe 32 - CAAD Education - Volume 1
through historic photographs and future proposed
designs (fig.7). The Highline app utilizes a Layar AR
browser that is available for most mobile platforms.
After initial time spent on understanding Layar SDK
environment, students focused on gathering geo-
location data for individual points of interest (POIs)
(fig.8) and setting up online databases.
Since this particular section of the course was
made up almost exclusively of architecture and de-
sign students, teaching faculty had to provide ini-
tial help with basic PHP programming and MySQL
database setup. As part of the development process,
students participated in hands-on workshops orga-
nized by teaching faculty and on some occasions re-
ceived a skeletal prototype of an app. This helped
to stage the progress of the project in such a way
that at any level of its development, students had a
fully functional prototype ready for testing with vari-
ous numbers of features and assets. The focus of the
student design team was on gathering relevant infor-
mation, imagery, and outside references. The second
stage involved population of the database, interface
design, and development of Web page links with ad-
ditional information.
Since many of the assets were Web pages related
to the app content, students had to consider designs
that were both desktop and mobile browser friendly.
This quickly became a challenge on its own, consid-
ering the diversity of mobile devices (phones and
tablets), their screen resolutions, and horizontal/ver-
tical layouts. Once all the assets were in place, the
design team focused on overall packaging, user ex-
perience, and interface design.
Figure 6
Highline
navigational AR app
Figure 7
The diversity of
assets developed
by students
Figure 8
Points of Interest
(POIs) for Highline
app
CAAD Education - Volume 1 - eCAADe 32 |361
Augmented Interiors
The goal of the project was to enhance communica-
tion between interior designers and their clients, and
to empower consumers to experience the impact a
particular design or set of furniture may have on their
home. Traditional home decorating is done by imag-
ining what a space would look like with the new fur-
niture or other design features without having a true
sense of scale or color gamut. Most commonly, cus-
tomers would measure the space in a house and see
if a new piece of furniture would fit within. Let's con-
sider another scenario.
You'relook ing througha furniture catalog and find
a piece that you like. But how would it fit into your liv-
ing room? Now you can find out without leaving your
couch, or wherever you are. Take the marker attached in
the catalog, place it on the desired location, download
the AR app, and look through the display of your mobile
device camera. The piece of furniture you're considering
is there for you to see in the context of your own living
room.
The marker-based AR application associates
each marker with a piece of furniture, material color,
or design features. The combination of markers al-
lows for a high number of variations of possible de-
signs. This app would allow ordinary people to take
design into their own hands and see exactly how a
new furnishing would look like in the context of their
home before they buy it. The approach does not re-
quire a digital rendering of the entire room. Instead,
it overlays a product in real time over the image of the
existing space. The applicability of AR in this project
is appropriate; it achieves its desired effect with very
few resources, could be easily commercialized, and
has the potential of reaching a broad consumer pop-
ulation.
This AR application was developed with the Pro-
cessing platform utilizing the ARToolKitlibrar y,which
provided a relatively streamlined production process
with good online support. The actual prototype
(fig.9) was developed for a regular PC laptop, not a
mobile device.
Figure 9
A student working
with a
marker-based AR
application
Figure 10
A student
interacting with the
Maze Game
362 |eCAADe 32 - CAAD Education - Volume 1
The major limitation of this platform was a low
resolution of AR markers-16 by 16 pixels-allowing for
a small number of possible marker variations. This
provided a major limitation to the size of the project
and the number of assets. Furthermore, even with
mid-size projects, some markers were easily mistaken
when they looked too similar to each other.
The Maze Game
This marker-based AR game involves navigation of a
virtual ball through a virtual maze by physically mov-
ing around and tilting the AR marker (fig.10). Move-
ments and adjustments of the marker in the physi-
cal world are translated to the virtual space and in-
terpreted with physics-based interaction (gravity and
collisions).
While this is a relatively straightforward design,
it involved a wide range of problems to be resolved
and thus multiple software toolkits. Four main toolk-
its that contribute to this game's functionality. FLAR-
toolkit (Flash/ActionScript port of ARToolkit) deals
with camera and marker detection. Papervision3D
(open-source real-time 3D engine for Flash) deals
with the construction and placement of maze walls.
JiglibFlash (open-source ActionScript 3D physics en-
gine) provides the collision detection between the
ball, the floor, and the maze walls as well as gravity to
propel ball movement. Finally, FlashDevelop (open-
source code editor for ActionScript 3) compiles all the
layers of code and runs the game application. The
maze walls are built and placed based on X, Y, and
Z coordinates, and then the gravity is directed in-
ward from the Z axis. The floor is the plane with colli-
sion detection, preventing the virtual ball from falling
down.
Another variation of this game, proposed but not
realized, could utilize a mobile device instead of a
computer. It would use a stationary marker with the
device functioning as a display and a virtual maze. In
this case, the device's accelerometer, compass, and
tilt sensors would provide the rotation and slop infor-
mation. This is one of the earlier projects developed
for the course before the Vuforia and Unity3D plat-
forms were introduced. Presently, the latter would be
the platform of choice from the physics engine, eas-
iness of development environment, and mobile out-
put perspectives. Additionally,the Unit y3D game en-
gine would provide a more effective and streamlined
environment for graphic user interface (GUI) devel-
opment, particularly in tracking game scores and en-
hancing user interactions.
CLASS OBSERVATIONS AND LESSONS
LEARNED
AR technology is entering a new stage where it is no
longer the domain of technology-oriented individ-
uals with heavy involvement of computer program-
ming and other software tools. Products such as Vu-
foria, Qualcomm's plugin for the Unity3D Game En-
gine, delivers a highly functional tool that can be
easily integrated into academic teaching and profes-
sional practice. The choice of a game engine like
Unity3D further makes the commercialization of AR
technology easier and more imminent. The ability
to integrate physics and other modules already exist-
ing in game engines simplifies the development pro-
cess and reduces the need for technology savviness
from the creative team. This does not mean that the
development is completely effortless as far as cod-
ing is considered-scripting is always required for ef-
fective game engine implementations-but it signifi-
cantly eases the learning curve, leading to democ-
ratization of digital creative tools. This transition
from technology heavily involving tools to designer-
oriented technology was visible in class projects with
over time, shifting from Processing and Flash-based
development environments (Interior Decoration and
3D Maze projects) to Vuforia and Unity3D. This di-
rectly facilitates the content and the user becom-
ing the primary drivers for the future of AR. This
also suggests that the climate is ready for design
schools to embrace AR technology as a new creative
and information visualization medium. Through the
AR projects and courses discussed above, students
are becoming aware of new modes of visual and
data-based thinking. Concepts such as location and
CAAD Education - Volume 1 - eCAADe 32 |363
context awareness form an important framework for
dealing with the over-supply of information and nav-
igating the current, almost ubiquitous data jungle.
While teaching AR-based courses, early lectures
and discussion are usually heavily involved with the
mechanics of AR technology, which often overwhelm
students. However, this initial technology shock
quickly evaporates, with projects' focus shifting to-
ward design, user experience, and content. Projects
connect with other disciplines and uses that respond
to a broad range of social and cultural needs. Stu-
dents perceive AR technology, even more than other
modes of computer graphics, as highly transpar-
ent, without a strong technological footprint. Thus,
this technology naturally transitions them to explore
diverse content-based topics. There was very lit-
tle "technology for the sake of technology" attitude
among students, who naturally gravitated toward
the multitude of ways to connect AR technology with
design, cultural, or social needs.
Furthermore, many of the team projects were de-
veloped by the multidisciplinary teams with students
from various programs across the campus, including
digital design, computer science, and information
technologies, but also architecture, interior design,
product design, and media communications. Some
of the most successful teams were in fact a random
combination of competent and passionate strangers
who just met during the first class when prospective
semester projects were discussed and teams were
formed. The fact that students brought to the team
their own distinct competencies and were able to re-
alize their expertise provided a successful base for
their team projects.
CONCLUSIONS
AR-based applications increasing occupy an impor-
tant place in branding/marketing, tourism, educa-
tion, and many other parts of life. AR has brought the
virtual and the physical world closer and made them
highly interconnected and interdependent through
location awareness, enhanced data overlays, and
user-focused content. It also finds its applications in
a diverse range of disciplines and ...
A number of the AR applications discussed here
exemplify an idea of "learning anytime, anywhere,"
which builds on Weiser's proposition for the role of
computation in the 21st century (Wesser, 1991). This
new role synergizes key characteristics of AR environ-
ments that include location awareness of data sets,
always-connected networks, and the ability to super-
impose images of the physical world with interactive
digital graphics. It allows for passive as well as ac-
tive interaction with information and virtual content.
Users are able not only to visually experience static
information but also to interact with data in more
dynamic and speculative ways by posing "what if..."
questions. These speculative investigationscreate an
environment of increased user engagement with the
benefits of experiential learning.
REFERENCES
KIM, K, SEO, B, HAN, J and PARK, J 2009 'Augmented Real-
ity Tour Systemfor Immersive Experience of Cultural
Heritage', In Proceedings of VRCAI, Yokohama, Japan
NIEDMERMAIR, S and FERSCHIN, P 2011a 'AnAugmented
Reality Framework for On-Site Visualization of Ar-
chaeological Data.',In Proceedings of the 16th Interna-
tional Conference on Cultural Heritage and New Tech-
nologies, http://www.stadtarchaeologie.at/?page_-
id=5665, pp. 636-647
NIEDMERMAIR, S and FERSCHIN, P 2011b 'An Aug-
mented Reality Framework for Architectural Appli-
cations', In Proceedings of the 8th International Sym-
posium on Location-Based Services, ed. Georg Gartner
and Felix Ortag,, pp. 192-205
WEISER, M 1991, 'The Computer for the Twenty-FirstCen-
tury', Scientific American, 1991, pp. 94-100
[1] www.metaio.com/customers/case-studies/lego/
[2] the-tuts.org
[3] www.layar.com/layers/pdnar1/
[4] comeoutandplay.org/2008_comfortofstrangers.php
[5] www.layar.com/layers/highlinefinal/
364 |eCAADe 32 - CAAD Education - Volume 1
