Content uploaded by Andrzej Zarzycki
Author content
All content in this area was uploaded by Andrzej Zarzycki on May 23, 2015
Content may be subject to copyright.
Augmenting Design Curriculum with Location-Aware
Technologies
A. Zarzycki
New Jersey Institute of Technology, Newark, New Jersey, USA
Abstract
This paper discusses ways in which emerging interactive augmented reality (AR) technologies are being
adopted by designers and extended into areas of tourism, education, entertainment and commerce. It
discusses, in detail, project development stages and methodologies used to engage design focused
students into, often complex, technological issues. The discussion is contextualized through a number of
case studies of mobile and marker-based AR applications developed within the university curriculum.
Categories: Human-centered computing - Human computer interaction (HCI) - Interaction paradigms -
Mixed / augmented reality
1. Introduction
Historically, new forms of representation and visualization
marked significant conceptual shifts in design thinking and
in the very process of ideation. Not unlike the introduction
of perspective during the Renaissance, digital technologies
in recent years transform the ways design is being
conceived and produced. Augmented reality (AR) is one of
the recent contributors to this shift, bringing significant
new attitudes into already complex digital processes.
Over the last couple of years, AR technology has
seen a renewed interest from developers and consumers as
well as the emergence of new authoring-friendly platforms
with a low learning curve. This new lower authoring
threshold attracted increasing interest from design
professionals and academics to port this new technology
into design and creative disciplines. From architecture and
product design to branding and gaming, AR technology
expands existing communication frameworks into more
user-centered media. The appeal of the AR technology
connects with the broader expectation of context-specific
and context-aware data sets that respond to user needs.
This paper investigates ways that emerging interactive
technologies are being adopted by designers and extended
into the areas of tourism, education, entertainment, and
commerce. It discusses in detail the project development
stages and methodologies used to engage designers into
often-complex technological issues. The discussion is
contextualized through a number of case studies of mobile
and marker-based AR applications developed as part of
academic research. These applications include an app for
fashion-based social events that allows participants to
preview recent collection additions, an info-navigational
app for the High Line elevated urban park in New York
City, a marker-based sustainable growth game, and an
interior decorating interface to visualize various furnishing
scenarios.
While a number of case studies are discussed from a
developer perspective touching on technical intricacies, the
primary focus is on content development, interface design,
and user interaction considerations.
2. Augmented Reality Context
Traditionally, AR environments employed two distinct
types of data overlay. Marker-based environments
employed distinct markers and, more recently, images
(image targets) to locate virtual data within the physical
world. Markers can be two-dimensional images or in some
cases three-dimensional captures/objects. A second,
marker-less, approach associated with mobile AR involved
GPS, digital compass, and accelerometer sensors to
position users and the virtual content around them. The
GPS location and the compass direction is checked against
a database for location associations. If any dataset point
meets user defined criteria, such as proximity or thematic
interests, the visual reference to the dataset is displayed on
a mobile device as a point of interest (POI).
3. Situated Past
While AR technology is routinely employed in the form of
data overlays providing supplementary information for
physical objects that are visible with the unaided eye, it is
also increasingly used to visualize less tangible structures
and concepts such as historical events, cultural phenomena,
and scientific processes. This can be seen in a number of
research projects and mobile apps developed as part of the
academic research, which help users to experience history
and facilitate urban explorations. [NF2011b] TimeWarp
[HBM*2008], a mobile edutainment application designed
as an AR game situated in Cologne, Germany, focuses on
virtual reconstruction of historic buildings by
superimposing virtual imagery over currently existing
structures. The application not only shows no-longer-
existing buildings as they originally appeared but also
visualizes design changes that occurred over time to still-
present structures.
An example of no-longer-existing buildings that
continue to function as an urban landmark are the Twin
Towers of the World Trade Center in New York. The 110
Stories1 mobile AR app virtually recreates the silhouette of
Twin Towers and presents them overlaid over camera
EUROGRAPHICS 2015/ M. Bronstein and M. Teschner Education Paper
c
The Eurographics Association 2015.
DOI: 10.2312/eged.20151024
images from various locations in the city (fig.1). Once the
most prominent landmark in the lower Manhattan, the
towers no longer serve as an element of wayfinding.
However, for those who want to reconnect with the lost
iconography of the towers, the 110 Stories app provides a
meaningful interface for the collective memory. Similarly,
a number of other AR apps bring to life past events or no-
longer-existing buildings and cities. These integrated media
elements may no longer be universally recognizable or
understood by the community nor contribute to universally
shared collective memories of a place.
Figure 1, Manhattan skyline as viewed with 110 Stories
mobile AR app.
Similarly, the Immersive Experience of Cultural
Heritage project [KSH*2009] uses an AR tour approach to
provide tourists with a more realistic experience by placing
virtual characters within historical structures. Visitors to the
heritage sites of Sajeongjeon and Gangnyeongjeon in
Korea can use their mobile devices to access additional
facts associated with the showcased physical content.
The Augmented Reality Framework for Architectural
Applications project [NF2011a] combines dynamic image
tracking of architectural context as a spatial framework for
historical reconstruction. While a similar approach is
routinely used by many museums, this particular project
does not rely on AR markers such as QR codes. It
implements visual camera tracking of the rectangular
display space to position its virtual actors without a need
for visually intrusive markers.
Virtual environment allow for explorations of inaccessible
or not-yet-materialized designs. They can be precursors of
future physical urban spaces and potent drives in their
realization. This is the case with AR environments (fig.2)
developed by Tremont Underground Theater Space (TUTS)
initiative2. This initiative is using AR gamified virtual
tourism media not only to popularize ideas of the adaptive
reuse of the abandoned public infrastructure but also to
build social constituency and connect with general public
(fig.3).
The shifting focus from virtual-reality (VR)
environments toward mixed-reality and AR frameworks
indicates the reexamination of earlier visions of separated
physical and digital worlds. The emerging picture fuses
both dimensions into a single continuum. The newfound
physical context adopted by AR games encourages players
to push the boundaries of social conventions and accepted
public behavior. Unlike more passive forms of
entertainment such as reality TV or even active-yet-
confined console-based games, the AR framework
incorporates physical activities and social interaction as
well as encouraging exploration, learning, and discovery.
Furthermore, as activities integrate digital media culture
within the built environment—cities—these games provide
an insight into our physical-digital selves and better
understanding of ourselves and our communities.
Figure 2. Augmented reality (AR) environment as social
and design activism and urban games.
Figure 3. Mystery Spaces, a map with POIs arranged in
the form of the game play.
4. Engaging Commerce and Tourism
“Let’s pretend you’re on your way to Manhattan to
buy some new clothes. Maybe you’re looking to
impress someone on a date. Maybe you need an
outfit to ace that interview. Maybe you’re looking to
change your style and try something new. Whatever
the case may be, you know New York fashion will not
disappoint.
You arrive on 54th Street on Fifth Avenue early
in the afternoon. There’s a ton of different stores in
that five-block radius. There’s high end fashion
retail and some typical name brand stores. Some
stores aren’t in your price range, but you might be
interested in what they have to offer. You’re not sure
where to shop first. Decisions, decisions…Luckily,
there’s an app for that!”
c
The Eurographics Association 2015.
A. Zarzycki / Augmenting Design Curriculum with Location-Aware Technologies32
The above shows one of the proposals for a mobile AR
application that combines fashion retail experience with
location-aware services and electronic social networks.
This application not only addresses an interest match
between product providers and consumers but also touches
on less tangible needs such as the sense of social happening
and the excitement associated with emerging technologies.
The FashNYC app is geared toward those interested
primarily in high-end fashion designers. It helps its users
make smarter shopping choices while connecting with the
current fashion scene. FashNYC brings awareness to the
fashion industry, connects with seasonal events, creates an
exciting new shopping experience, and establishes a
presence in today’s heightened mobile application culture.
Through the app, consumers have access to online videos
showcasing each brand’s current collections. They can
watch interviews with designers and access garment
information including sizes, colors, materials, prices, and
availability. What is particularly interesting about this
proposal is its narrow, yet integrated, approach to providing
services. It identifies a specific target group of potential
users and attempts to deliver a comprehensive solution
reminiscent of the traditional one-stop shopping approach.
The app was developed for the Layar AR browser
using PHP programming with MySQL database. The
student team focused not only on creation of individual
assets and associated Web pages (fig.4) but also on the
overall packaging, user experience (UX) (fig.5), and final
layout for the navigational map (fig.6). The idea of
branding, characteristic of the fashion and design industry,
was central when developing the user interface (UI). Since
the seamless connectivity to various social networks and
fashion-based websites was key here, the app UI became a
critical part of the overall effort.
Figure 4. Virtual changing room with FashNYC
Figure 5. Mobile interface for FashNYC
Figure 6. App navigation map
While the student team developed a fully functional
prototype,3 they also proposed the next level of
functionality that went beyond the scope of the class and
their technical competencies. This was an important part of
the overall design strategy, where each team developed the
initial project concept and later expanded it into a more
formalized business proposal with outlined future
development stages. This became an important learning
experience about technology development and idea
implementation. The ability to project a series of
intermediate goals with each phase forming a defined and
comprehensive package that achieves particular
functionalities helped students to divide work based on
their expertise. At the same time, it helped in setting
realistic and easily achievable goals even within a relatively
short course timeline. One of the proposed features that
could be implemented with Layar’s interactive graphics
was garment (image) recognition. Another was developing
a screening tool that would provide shopping alternatives.
The proposed app functionality was expressed in the
following statement.
Let’s say you’re looking for a new blue suit. You see
one you like in the window of Versace. To be quite
frank, you can’t afford it, but you really like the color
and the style. You can take a picture of the suit and
have the application search for it. As the application
searches for it, you take the time to watch a video of
the Fall/Winter 2013 runway show where the suit was
first featured. You notice you like the way it looks
when on the runway, but the blue one that was shown
looks even better. Once the suit has been found in the
search, you can try to find similar garments in nearby
stores. You’re in luck! That application located a
c
The Eurographics Association 2015.
A. Zarzycki / Augmenting Design Curriculum with Location-Aware Technologies 33
similar blue tight-fitting suit in Zara across the street
for half the price! Bam! Success. (Philip Caleja,
Nicholas Haby, and Daniel Schittone)
While this particular functionality was never implemented,
it provided a stimulating basis for discussion on the ways it
could be realized in the future. Two approaches were
identified: (1) as a searching application/agent to find
alternatives based on the combination of text and image
recognitions, and (2) as a method of capitalizing on user
purchasing/searching patterns similar to Amazon’s
“Customers Who Bought This Item Also Bought,” which
suggests additional purchases to customers.
As with any new app or a product relying on social
interconnectivity and input, the key issue is to develop a
critical mass of active users who would propagate its
virtual life. This is a major challenge facing many new
media products including an AR community like the one
proposed by students. The strategy to address this
impediment and help with the future commercialization of
the app was to tie it to a particular event that is highly
localized with a defined time frame. The student team
proposed to connect it to the Fashion’s Night Out (FNO)
event or the New York Fashion Week. While this was not
implemented yet, it provides a feasible strategy for
launching an AR app product that is highly contextualized
with its theme, location, and timing. This is also an
approach used by other mobile AR games such as Comfort
of Strangers4 that rely on a critical mass of participants for
their success. More recently, an augmented reality
massively multiplayer online role playing game
(MMORPG) Ingress5 used a similar approach of staged
events/happenings to develop momentum and mobilize its
players.
Highline Tour6 is a navigational and informational mobile
AR app geared toward visitors to the High Line, an urban
park in New York City (fig.7). It provides users with
historical and current information as well as plans for
future developments. Its location-aware functionality
allows for sorting and positioning data in relationship to the
urban context. It shows year-around activities with imagery
of various plants and foliage reflecting seasonal changes
occurring in the park. In many aspects the app functions as
a time capsule that combines multiple layers of information
into a single geo-location. These multiple layers can be
individually accessed and combined to provide a selected
perspective into the High Line project. Users of the app can
look at a particular section of a project and freely navigate
through historic photographs and future proposed designs
(fig.8). To some extent this media overlay provides a third
alternative to “renovate and loose the charm of the past”
versus “keep the past untouched and do not adapt to new
uses or current needs.” The AR component, at least
virtually, preserves to a certain extent the original
conditions and memories of the past.
The Highline app utilizes a Layar AR browser that is
available for most mobile platforms. After initial time spent
on understanding Layar SDK environment, students
focused on gathering geo-location data for individual points
of interest (POIs) (fig.9) and setting up online databases.
Since this particular section of the course was made up
almost exclusively of architecture and design students,
teaching faculty had to provide initial help with basic PHP
programming and MySQL database setup.
Figure 7. Highline navigational AR app
Figure 8. The diversity of assets developed by students
Figure 9. Points of Interest (POIs) for Highline app
As part of the development process, students
participated in hands-on workshops organized by faculty
and on some occasions received a skeletal prototype of an
app. This helped to stage the progress of the project in such
a way that at any level of its development, students had a
fully functional prototype ready for testing with various
numbers of features and assets. The focus of the student
design team was on gathering relevant information,
imagery, and outside references. The second stage involved
population of the database, interface design, and
c
The Eurographics Association 2015.
A. Zarzycki / Augmenting Design Curriculum with Location-Aware Technologies34
development of Web page links with additional
information. Since many of the assets were Web pages
related to the app content, students had to consider designs
that were both desktop and mobile browser friendly. This
quickly became a challenge on its own, considering the
diversity of mobile devices (phones and tablets), their
screen resolutions, and horizontal/vertical layouts. Once all
the assets were in place, the design team focused on overall
packaging, user experience, and interface design.
While the AR app development process required
a broad set of skills, from basic programming to graphic
design, this diversity made students more engaged in the
project and provided opportunities for the development of
specific expertise with a relatively low learning curve.
5. Contextualized Learning
The design process can be seen as developing multiple
competing spatial scenarios and testing them against
logistical constraints. The Sustainable Growth Economy
game follows the same framework and uses AR marker-
based mobile technology to provide an immediate visual
feedback and greater participant engagement to players.
The game was developed as an educational aid for middle
school students to understand the consequences of various
economic growth choices and the fine balancing required to
maintain economic growth. The game mechanics parallel a
simplified version of the SimCity model, with a focus on
increasing the visual feedback loop as an important part of
the game play. The goal of the game is to grow a
civilization that is self-sustaining. This requires addressing
the following three competing objectives: (1) reaching a
certain level of population, (2) maintaining a habitable
environment, and (3) sustaining enough energy to keep the
civilization going. Each of these objectives directly
compromises the remaining two, which requires a fine
balancing act. The game play relies on the use of physical
cards (markers/image targets; see fig. 10), which facilitate
virtual community growth. Players use these cards to add
and remove assets or to modify their properties. The
outcomes of the game play are registered on mobile devices
(fig. 11), which provide an opportunity for open—what-
you-see-is-what-you-get (WYSIWYG)—or individualized
game maps. Since most of the assets are virtualized (visible
only on the screen), they can be easily customizable and
tailored based on the player preferences and the experience
level. While this game focuses on the sustainable growth
economy, its dynamics and mechanics could be adopted to
serve as town master planning or building programming
tools, helping designers to engage clients and the broader
public. In this case, the game fulfills the role of the design
facilitator, and to some extent design simulator, with an
immediate and visually explicit feedback.
Figure 10. AR markers used as game cards.
Figure 11. An early prototype of the game with assets
visible on the mobile screen
6. Augmented Interiors
The goal of the project was to enhance communication
between interior designers and their clients, and to
empower consumers to experience the impact a particular
design or set of furniture may have on their home (fig.12).
Traditional home decorating is done by imagining what a
space would look like with the new furniture or other
design features without having a true sense of scale or color
gamut. Most commonly, customers would measure the
space in a house and see if a new piece of furniture would
fit within. Let’s consider another scenario.
You’re looking through a furniture catalog and find a
piece that you like. But how would it fit into your
living room? Now you can find out without leaving
your couch, or wherever you are. Take the marker
attached in the catalog, place it on the desired
location, download the AR app, and look through the
display of your mobile device camera. The piece of
furniture you’re considering is there for you to see in
the context of your own living room.
Figure 12. A student working with a marker-based AR
application
The marker-based AR application associates each marker
with a piece of furniture, material color, or design features.
The combination of markers allows for a high number of
variations of possible designs. This app would allow
ordinary people to take design into their own hands and see
exactly how a new furnishing would look like in the
c
The Eurographics Association 2015.
A. Zarzycki / Augmenting Design Curriculum with Location-Aware Technologies 35
context of their home before they buy it. The approach does
not require a digital rendering of the entire room. Instead, it
overlays a product in real time over the camera image of
the existing space. The applicability of AR in this project
is appropriate; it achieves its desired effect with very few
resources, could be easily commercialized, and has the
potential of reaching a broad consumer population. The
limiting factor of this particular project was its
development environment—Processing with ARToolkit.
While Processing is an excellent designer prototyping tool,
the AR extension had a series of limitations in the number
and resolution of fiduciary markers. This limitation can
presently be significantly overcome with applications such
as the Vuforia extension to Unity3D.
7. The Maze Game
This marker-based AR game involves navigation of a
virtual ball through a virtual maze by physically moving
around and tilting the AR marker (fig.10). Movements and
adjustments of the marker in the physical world are
registered in the virtual space and interpreted with physics-
based interaction (gravity and collisions).
Figure 9. A student interacting with the Maze Game
While this is a relatively straightforward design, it
involved a wide range of problems to be resolved and thus
multiple software toolkits. Four main toolkits that
contribute to this game’s functionality. FLARtoolkit
(Flash/ActionScript port of ARToolkit) deals with camera
and marker detection. Papervision3D (open-source real-
time 3D engine for Flash) deals with the construction and
placement of maze walls. JiglibFlash (open-source
ActionScript 3D physics engine) provides the collision
detection between the ball, the floor, and the maze walls as
well as gravity to propel ball movement. Finally,
FlashDevelop (open-source code editor for ActionScript 3)
compiles all the layers of code and runs the game
application. The maze walls are built and placed based on
X, Y, and Z coordinates, and then the gravity is directed
inward from the Z axis. The floor is the plane with collision
detection, preventing the virtual ball from falling down.
Another variation of this game, proposed but not
realized, could utilize a mobile device instead of a
computer. It would use a stationary marker with the device
functioning as a display and a virtual maze. In this case, the
device’s accelerometer, compass, and tilt sensors would
provide the rotation and slop information. This is one of the
earlier projects developed for the course before the Vuforia
and Unity3D platforms were introduced. Presently, the
latter would be the platform of choice from the physics
engine, easiness of development environment, and mobile
output perspectives. Additionally, the Unity3D game
engine would provide a more effective and streamlined
environment for graphic user interface (GUI) development,
particularly in tracking game scores and enhancing user
interactions.
8. Broader Discussion
AR technology is entering a new stage where it is no longer
the domain of technology-oriented individuals with heavy
involvement of computer programming and other software
tools. Products such as Vuforia, Qualcomm’s plugin for the
Unity3D Game Engine, delivers a highly functional tool
that can be easily integrated into academic teaching and
professional practice. The choice of a game engine like
Unity3D, used to develop Sustainable Economy game,
further makes the commercialization of AR technology
easier and more imminent. The ability to integrate physics
and other modules already existing in game engines
simplifies the development process and reduces the need
for technology savviness from the creative team. This does
not mean that the development is completely effortless as
far as coding is considered—scripting is always required
for effective game engine implementations—but it
significantly eases the learning curve, leading to
democratization of digital creative tools. This transition
from technology heavily involving tools to designer-
oriented technology directly facilitates the content and the
user becoming the primary drivers for the future of AR.
This also suggests that the climate is ready for design
schools and practices to embrace AR technology as a new
creative and information visualization medium.
A number of the AR applications discussed here
exemplify an idea of “learning anytime, anywhere,” which
builds on Weiser’s proposition for the role of computation
in the 21st century [Wes1991]. This new role synergizes
key characteristics of AR environments that include
location awareness of data sets, always-connected
networks, and the ability to superimpose images of the
physical world with interactive digital graphics. It allows
for passive as well as active interaction with information
and virtual content. Users are able not only to visually
experience static information but also to interact with data
in more dynamic and speculative ways by posing “what
if…” questions. These speculative investigations create an
environment of increased user engagement with the
benefits of experiential learning.
c
The Eurographics Association 2015.
A. Zarzycki / Augmenting Design Curriculum with Location-Aware Technologies36
Through the AR projects and courses discussed above,
students are becoming aware of new modes of visual and
data-based thinking. Concepts such as location- and
context-awareness form an important framework for
dealing with the over-supply of information and navigating
the current, almost ubiquitous data jungle.
While teaching AR-based courses, lectures and
discussion are usually heavily involved with the mechanics
of AR technology, which often overwhelm students this
initial technology shock quickly evaporates, with projects’
focus shifting toward design, user experience, and content.
Projects connect with other disciplines and uses that
respond to a broad range of social and cultural needs.
Students perceive AR technology, even more than other
modes of computer graphics, as highly transparent, without
a strong technological footprint. Thus, this technology
naturally transitions them to explore diverse content-based
topics. There was very little “technology for the sake of
technology” attitude among students, who naturally
gravitated toward the multitude of ways to connect AR
technology with design, cultural, or social needs.
The course discussed above attracted a diverse
group of students. While all of them were interested in the
AR technology, they all saw different opportunities
emerging from its use. For architecture students it allowed
for an interactive visualization of buildings and built
environments with context-aware capabilities. Similarly,
industrial design majors perceived AR technology as an
effective way to communicate product design, its assembly,
and product branding. A number of information technology
(IT) students developed gaming applications, while
communication students saw opportunities in various
aspects of interactive print and gamification-based
motivational activities. Ultimately the AR technology
became a lateral platform for collaboration that manifested
itself in a diverse range of applications, with each of them
well integrated into and benefiting individual design and
communications disciplines.
9. Conclusions
AR-based applications increasingly occupy an important
place in branding/marketing, tourism, education, and many
other parts of life. AR has brought the virtual and the
physical world closer and made them highly interconnected
and interdependent through location-awareness, enhanced
data overlays, and user-focused content. It also finds its
applications in a diverse range of disciplines. This is
evident in the types of applications coming to the mobile
market and in the technology development associated with
Google Glass, Microsoft’s HoloLens, and even products
like Google Cardboard. All these indicate broad and rapidly
developing interests that are also reflected in academic
curricula and general public attitudes.
References
[HBM*2008] HERBST A., BRAUN K., McCALL R. &
BROLL W.: TimeWarp: interactive time travel with a
mobile mixed reality game. (2008) Pages 235–244 in
Proceedings of MobileHCI 2008. ACM Press,
Amsterdam.
[NF2011a] NIEDMERMAIR, S., FERSCHIN, P.: An
Augmented Reality Framework for On-Site
Visualization of Archaeological Data. In Proceedings of
the 16th International Conference on Cultural Heritage
and New Technologies, 636-647. 2011. Museen der Stadt
Wien–Stadtarchäologie.
http://www.stadtarchaeologie.at/?page_id=5665.
[NF2011b] NIEDMERMAIR, S., FERSCHIN, P.: An
Augmented Reality Framework for Architectural
Applications. 2011. In Proceedings of the 8th
International Symposium on Location-Based Services,
ed. Georg Gartner and Felix Ortag, 192–205.
[KSH*2009] KIM K., SEO, B., HAN, J. & Park, J.:
Augmented Reality Tour System for Immersive
Experience of Cultural Heritage (2009). In Proceedings
of VRCAI 2009, Yokohama, Japan, December, 2009.
[Wes1991] WEISER M., The Computer for the Twenty-
First Century (1991). Scientific American, pp. 94–10,
September 1991.
Student Project Credits
FashNYC: Philip Caleja, Nicholas Haby, and Daniel
Schittone
Highline Tour: Mike Litus, Pawel Zawistowski, and Travis
Flick
Augmented Interiors: Samantha Goldman and Kirstianne
Marcado
The Maze Game: David Einstein
The Sustainable Growth Economy: Louis Saporito and
James Wolff
Endnotes
1 www.110stories.com/
2 the-tuts.org
3 www.layar.com/layers/pdnar1/
4 comeoutandplay.org/2008_comfortofstrangers.php
5 https://www.ingress.com/
6 www.layar.com/layers/highlinefinal/
c
The Eurographics Association 2015.
A. Zarzycki / Augmenting Design Curriculum with Location-Aware Technologies 37
