Content uploaded by Johannes Schöning
Author content
All content in this area was uploaded by Johannes Schöning on Sep 11, 2017
Content may be subject to copyright.
The Madeira Touch: Encouraging
Visual-Spatial Exploration using a
Tactile Interactive Display
Abstract
The current information marketplace for tourists is
dominated by for-profit purveyors of information.
Potential visitors must rely on experts-for-hire or
search engine results in order to learn about a desired
destination. In this paper, we introduce The Madeira
Touch, a multimodal display installation rooted in the
unique characteristics of Madeira, which allows users to
explore the island by selecting a type of scenery and
showing the user-generated photos of that type of
scenery in a map-based interface. To make this
pervasive display more engaging, we designed an
exploratory tactile-input mode of interaction: users will
be able to touch a physical object, representing a type
of scenery (a rock for mountains, a seashell for the sea,
etc.), which will then bring up suitable photos of that
type of scenery overlaid on a map of the island. The
display will help users to form their mental image of the
island and to plan trips that best suit their interests.
Author Keywords
Pervasive display, tactile interaction, multimodal
interaction, user-generated content, digital signage
ACM Classification Keywords
H.5.m. Information interfaces and presentation (e.g.,
HCI): Miscellaneous.
Copyright is held by the author/owner(s).
CHItaly ’17, September 18-20, 2017, Cagliari, Italy.
Catia Prandi
ARDITI, Madeira-ITI
Funchal 9020-105, Portugal
catia.prandi@m-iti.org
Catherine Chiodo
Ricjeareu Villaflor
Carnegie Mellon University
Pittsburgh PA 15213, USA
catherine@chiodo.org
rj.villaflor@gmail.com
Nicolas Autzen
Johannes Schöning
University of Bremen
Bremen 28359, Germany
n.autzen@uni-bremen.de
schoening@uni-bremen.de
Introduction & Motivation
In an increasingly connected world, travelers seeking
local experiences may encounter a paradox. While it is
easy to find information about destinations and places
that they may wish to visit, it is also increasingly
difficult to differentiate between the many options
available. At the same time, travelers who have
witnessed the homogenizing effects of globalization
may place a particular premium on unique experiences
that can only be found in certain places.
In this project, we took the island of Madeira as a
representative case study of a well-known touristic
destination that could be made more discoverable to
the island’s visitors. However, while tourists may
choose to visit Madeira because of its beautiful scenery
and outdoor activities, often related to the island’s
levadas, remote canals that serve as walking paths,
they may not know what destinations on the island are
best suited to the kinds of sceneries they hope to see.
Currently, travelers who hope to experience natural
beauty can plan their trips by either starting with a
possible location and attempting to find correlating
photos, or by beginning with photos of destinations
they would like to visit and then attempting to find
location information. Both approaches suffer
breakdowns when photos are not tagged with the
commonly-used location names.
In the field of pervasive display systems [4], we
designed our solution to investigate how the use of an
exploratory tactile mode of interaction to provide
georeferenced visual information in the form of user-
generated photos of points of interest (POIs) can
facilitate visitors in discovering locations on the touristic
destination, enhancing their experience. The result is
The Madeira Touch, a pervasive display that allows
users to correlate photos with locations, using a
multimodal interaction that allow users to select either
a type of scenery (touching a physical object) or a
location (using the map-based visualization). Madeira
represents an ideal place to develop a new way for
visitors to explore the island, however, it is our
intention for this system to be adaptable to other
tourist destinations, levering on the location’s unique
offerings and characteristics.
Figure 1: The Madeira Touch in context
The Madeira Touch
Our solution is to provide the Madeira’s main tourism
office, with our pervasive display. There, visitors to the
island will be able to explore a map of Madeira using
two modes of interaction: 1. traditional map-based
touchscreen interaction (Figure 2), or 2. exploratory
tactile-input interaction (Figure 3). With the first type of
interaction, visitors will be able to touch the digital map
and see user-generated photos of that location. With
the second one, users will be able to touch a physical
Scenarios of Use
Scenario 1: While waiting at
the tourism office, a visitor
notices a display with boxes
containing different objects
surrounding it, in front of the
entrance. She touches one
object and notices the display
showing images from a
levada walk in Madeira. She
removes her hand from the
object and the display returns
to the map of the island. She
continues to touch the
objects and see the images
associated with them. She
leaves the display with a
better understanding of the
opportunities on the island.
Scenario 2: A local Madeiran
visits the tourism office to
see the new interactive
display. While interacting
with it, he notices that one of
the photos is an image he
took on Instagram. He feels
gratified that one of his
images is contributing to the
experience available on the
display and to the information
provided to tourists on the
island.
object that corresponds to a type of scenery on the
island, which will then bring up user-generated photos
of that type of scenery (Figure 1). By enabling both
forms of interaction, we intend to encourage
exploratory scenery discovery as well as practical trip-
planning in context in which multiple users, such as
family groups, can co-experience the display [5].
Design Concepts
Traditional tourism information relies on professionally-
produced content which is limited both in quantity and
in coverage. We intend for this system to serve as an
exploration of how UGC from social media can be
curated to provide dynamic, updated and custom sets
of information for specific audiences (i.e., visitors).
While interactive displays have a high potential to
engage passersby, they frequently go unnoticed and
unused [7, 11], confirming the so called ‘display
blindness’ effect [8]. By situating our display in a
strategic location (i.e., in front of the entrance) of a
tourism office where visitors often wait to speak to
someone, we intend to mitigate this issue, exposing
visitors to the display at a time when they will be
inclined to investigate. To further increase visitor
engagement, we have incorporated a novel form of
input in the form of physical objects that a user may
touch to experience certain kinds of sceneries. In fact,
studies of initial engagement with interactive displays
have found that physical interactions prompt greater
rates of engagement among passersby [6]. Moreover,
our solution aims to overcome the ‘interaction
blindness’ [10] that often plagues public displays by
providing users with novel and suggestive physical
objects that encourage non-linear exploration. This
paradigm encourages a very different kind of
interaction with the data, engendering an experience
that is less goal-oriented and more exploratory. On the
other hand, the use of tactile physical objects can raise
the ‘affordance blindness’ issue, defined as the inability
to understand the interaction modalities of a public
display [3]. Our solution aims to moderate this
problematic providing visual hints to attract the user’s
attention (as described in Scenario 3).
Figure 2: Users can touch the thumbnails on the screen.
Figure 3: Users can directly touch the physical objects.
Implementation
The Madeira Touch software architecture is composed
of three main modules (as shown in Figure 4). The
Geotagged Photos Retrieval Module collects geotagged
photos and paths related to the main touristic
georeferenced pedestrian walks that characterize the
island. This stage, we have decided to use
OpenStreetMap (OSM), an open source system that
allows users to voluntarily collect and share GPS tracks
and georeferenced data (i.e. Point of Interests). In
Scenario 3: While his
parents are in the queue, a
15-year-old notices a monitor
surrounded by different
natural objects in display
boxes, in front of the
entrance. He tries to touch a
location on the monitor,
expecting a touch-based
interaction with the system.
In response to his touch, the
display shows a photo of that
area. At the same time, a box
lights up, grabbing his
attention. He notices that the
box holds a tree branch and
that the photo on the screen
is full of trees. Intrigued, he
decides to touch the tree
branch and the monitor
begins to display photos of
forests in different areas of
the island. He touches
another object, then another.
When his parents are done,
he brings them over to the
display and together they
explore the island.
Madeira, an island with an area of 802 km², the OSM
dataset includes 16000 points and 24000 lines along
with walking paths (levadas). There are also private
datasets gathered by companies that have collected
GPS tracks specifically related to walking paths such as
Walk Me Madeira (http://www.walkmeguide.com/).
Based on this dataset of GPS tracks and routes, the
module retrieves public geotagged photos from
different social media platforms and photo blogs to
continually integrate UGC (i.e. photos) to enrich the
user experience of our system [12]. At this stage, the
system includes the Instagram and Flickr platforms,
both of which provide developers with APIs for
retrieving public photos based on locations and/or tags.
The Classification Module is the core of our system,
because it enables (i) the elimination of photos which
include faces and other non-nature showing images and
(ii) the categorization of the collected pictures based on
the main objects (rocks, sand, etc.) and scenery (cliff,
forest, etc.) in each image. Each of these categories is
correlated with a physical object with which the users
can interact. Different kinds of machine learning
algorithms have been developed for the recognition and
classification of faces/landscapes/nature elements
represented in photos from social media platforms and
photo blogs [1, 2, 9, 13]. However, this system could
also use a crowdsourcing approach to let participating
users manually check the photos. A third approach
would be a system like the Google ReCAPTCHA, which
asks users to solve a puzzle by selecting all the images
that represent a specific element.
The Visualization Module is the final step in this system
and connects the categorized geotagged photos with
the object the user touches, showing the information in
a Google Maps based interface. Regarding the hardware
requirements, The Madeira Touch utilizes a touch
screen monitor and sensors to indicate when an object
has been touched. Considering the design of our
system, simple motion sensors should work well. The
current design also uses LED lights in each box to
emphasize the way in which each object corresponds to
a certain type of photo. This way, even when the user
is interacting directly with the touchscreen, the relevant
object will light up, indicating the relationship between
object and image.
Conclusion and Future Work
Our concept combines three characteristics in a unique
way to make an engaging pervasive display for visitors
to the island of Madeira. By using images from social
media, the system insures that the content remains
dynamic and accurate. By allowing users to explore the
data, either by location or by scenery type, the system
allows for scenery exploration and practical trip
planning. Finally, by creating a tactile form of
interaction, the system encourages users to consider
the natural materials represented, allowing them to
form a more sensual and complete mental picture of
the destination. While all three of these characteristics
are transferable and could be applied to other
destinations, it is the authors’ belief that, for future
installations of this solution, the appeal and utility of
such a system relies on its ability to accurately reflect
the unique character of the local environment. To
evaluate the effectiveness of our pervasive display in
engaging users and enhancing their visiting experience,
overcoming the display/interaction/affordance blindness
issues, we plan to install the system in the Madeira’s
main tourism office.
Figure 4: System architecture
References
1. Nuttapoom Amornpashara, Yutaka Arakawa,
Morihiko Tamai and Keiichi Yasumoto. 2015.
Landscape photo classification mechanism for
context-aware photography support system, In
Proceedings of Conference on Consumer Electronics
(ICCE 2015), 663-666.
http://doi.acm.org/10.1109/ICCE.2015.7066570
2. Pu Cheng and Jie Zhou. 2011. Automatic Season
Classification of Outdoor Photos, In Proceedings of
the Conference on Intelligent Human-Machine
Systems and Cybernetics, 46-49.
http://doi.acm.org/10.1109/IHMSC.2011.18
3. Jorgos Coenen, Sandy Claes, and Andrew Vande
Moere. 2017. The concurrent use of touch and mid-
air gestures or floor mat interaction on a public
display. In Proceedings of the Symposium on
Pervasive Displays (PerDis '17). Article 9, 9 pages.
DOI: https://doi.org/10.1145/3078810.3078819
4. Nigel Davies, Sarah Clinch, and Florian Alt. 2014.
Pervasive displays: understanding the future of
digital signage. Synthesis Lectures on Mobile and
Pervasive Computing 8.1 (2014): 1-128.
5. Jodi Forlizzi and Katja Battarbee. 2004.
Understanding experience in interactive systems.
In Proceedings of the conference on Designing
interactive systems: processes, practices, methods,
and techniques (DIS '04), 261-268.
http://doi.acm.org/10.1145/1013115.1013152
6. Wendy Ju and David Sirkin. 2010. Animate objects:
how physical motion encourages public interaction.
In Proceedings of the conference on Persuasive
Technology (PERSUASIVE'10), 40-51.
http://doi.acm.org/10.1007/978-3-642-13226-1_6
7. Kazjon Grace, Rainer Wasinger, Christopher Ackad,
Anthony Collins, Oliver Dawson, Richard Gluga,
Judy Kay, and Martin Tomitsch. 2013. Conveying
interactivity at an interactive public information
display. In Proceedings of the Symposium on
Pervasive Displays (PerDis '13), 19-24.
http://doi.acm.org/10.1145/2491568.2491573
8. Jörg Müller, Dennis Wilmsmann, Juliane Exeler,
Markus Buzeck, Albrecht Schmidt, Tim Jay, and
Antonio Krüger. 2009. Display blindness: The effect
of expectations on attention towards digital
signage. Pervasive Computing (2009): 1-8.
http://dx.doi.org/10.1007/978-3-642-01516-8
9. Mor Naaman, Susumu Harada, QianYing Wang,
Hector Garcia-Molina, and Andreas Paepcke. 2004.
Context data in geo-referenced digital photo
collections. In Proceedings of the conference on
Multimedia (MULTIMEDIA '04), 196-203.
http://dx.doi.org/10.1145/1027527.1027573
10. Gonzalo Parra, Joris Klerkx, and Erik Duval. 2014.
Understanding Engagement with Interactive Public
Displays: an Awareness Campaign in the Wild. In
Proceedings of the Symposium on Pervasive
Displays (PerDis '14), 180-186.
http://dx.doi.org/10.1145/2611009.2611020
11. Peter Peltonen, Esko Kurvinen, Antti Salovaara,
Giulio Jacucci, Tommi Ilmonen, John Evans, Antti
Oulasvirta, Petri Saarikko. 2008. It’s mine, don't
touch!: interactions at a large multi-touch display
in a city centre. In Proceedings of the Conference
on Human Factors in Computing Systems (CHI
'08), 1285–1294.
http://dx.doi.org/10.1145/1357054.1357255
12. Pavel Serdyukov, Vanessa Murdock, and Roelof van
Zwol. 2009. Placing flickr photos on a map.
In Proceedings of the conference on Research and
development in information retrieval (SIGIR '09),
484-491.
http://doi.acm.org/10.1145/1571941.1572025
13. Feng Tang, Daniel R. Tretter and Chris Willis. 2011.
Event classification for personal photo collections.
In Proceedings of Conference on Acoustics, Speech
and Signal Processing (ICASSP), 877-880.
http://doi.acm.org/10.1109/ICASSP.2011.5946544
