Content uploaded by Georgios Trichopoulos
Author content
All content in this area was uploaded by Georgios Trichopoulos on Sep 28, 2018
Content may be subject to copyright.
ARTISTS: A VIRTUAL REALITY CULTURAL EXPERIENCE
PERSONALIZED ARTWORKS SYSTEM: THE “CHILDREN
CONCERT” PAINTING CASE STUDY
George Trichopoulos1, John Aliprantis2, Markos Konstantakis3, George
Caridakis4
1Researcher, Department of Cultural Technology and Communication, University of
the Aegean, e-mail: gtricho@aegean.gr
2PhD Candidate, Department of Cultural Technology and Communication, University
of the Aegean, e-mail: jalip@aegean.gr
3PhD Candidate, Department of Cultural Technology and Communication, University
of the Aegean, e-mail: mkonstadakis@aegean.gr
4Assistant Professor, Department of Cultural Technology and Communication,
University of the Aegean, e-mail: gcari@aegean.gr
Abstract. In recent years, there is a constant tendency in integrating modern technologies into mobile
guides and applications in Cultural Heritage (CH) domain, aiming in enriching cultural user
experience. Amongst them, Virtual Reality (VR) has widely been used in digital reconstruction or
restoration of damaged cultural artifacts and monuments, allowing a deeper perception in their
characteristics and unique history. This work presents a VR environment that takes into account the
diverse needs and characteristics of visitors and digitally immerses them into paintings, giving them the
ability to directly interact with their characteristics with the Leap Motion controller. To test our
proposed system, a mobile prototype application has been designed, focused on the famous painting
“Children Concert” created by Georgios Iakovidis, which also integrates the User Personas and the
different scenarios depending on users’ profile.
Keywords: Cultural Heritage; Cultural User Experience; Natural Interaction; User personas; Virtual
Reality;
Introduction
In recent years, various works argue about the positive influence that Augmented
Reality (AR) and Virtual Reality (VR) could have on the fields of language studies,
social sciences, mathematics and physics, medical science, art, entertainment,
advertising and marketing (Chang Kuo-En, 2014). According to Chang, Chang, Hou,
Sung, Chao, Lee, VR and AR technologies promote art appreciation to museum
visitors during a visit. In other words, visitors that used those technologies to guide
through a museum learned more about the exhibits comparing to all other visitors that
used conventional guides (audio guides) or walked freely without any kind of
guidance. A VR guide can boost mental and visual focus on exhibits, achieving a
level of flow (Mihaly Csikszentmihalyi, 1975), which motivates user to seek more
knowledge and extend his visit.
Meanwhile, personalization methods in User Experience (UX) and Cultural User
Experience (CUX) appear to give a new perspective to mobile guides and applications
in Cultural Heritage (CH). Personalization (Antoniou & Lepouras, 2010) is based on
the assumption that an computer system can understand the user’s needs, while its
success relies greatly on the accurate elicitation of the user profile. The main reason
for personalization need is simple: Everyone is unique. Matching visitor’s experience,
knowledge and demands is a highly challenging and demanding task. Capturing
special personal characteristics, before or during the visit in a cultural site, has been
implemented using several methods, for example using ontologies, methodological
approach, statistical approach (Pujol Laia et al., 2012), or indirect approach by taking
advantage of social networks like Facebook (Antoniou Angeliki et al., 2016), or
finally, according to visitor’s age and behavior.
Current work presents a Virtual Reality interface that represents digitally the world of
paintings, allowing users to interact with the aspects of the painting in a 3D
environment. The presented framework also integrates personalization, user personas
(based on the User Personas methodology [Konstantakis Markos et al., 2017]) and
context awareness techniques to improve users’ experience. In Section 2 we briefly
present our ARTISTS framework, the technologies that we used along and how we
integrate them to the application, the frameworks’ architecture and a use case scenario
with our prototype based on and the famous painting “Children Concert” created by
Georgios Iakovidis. Finally in Section 3 we discuss our future work.
ARTISTS Framework
Description
ARTISTS is a mobile application that brings to life famous paintings, by digitally
construct its aspects in a Virtual Reality environment, where users can interact with its
3D models. Users immerse into the VR world by using their own devices mounted on
a VR headset (Google Cardboard), and then interact with the 3D environment using
gestures that are captured by the Leap Motion controller, that’s attached on the
headset. The proposed interface not only puts user inside a painting, allowing them to
observe and interact with the 3D models in many angles, but also uses various
methodologies (context-awareness, personalization, and gesture-recognition) in order
to enhance user’s cultural experience.
ARTISTS prototype has been designed based on the famous painting “Children’s
Concert” by Greek painter George Iakovidis, which can be found in Athens National
Gallery – Greece, and in a digital format in “George Iakovidis” digital gallery, in
Hidira village – Lesvos. For this painting, seven 3D human models were created,
along with their animations and sounds, in accordance with the 7 characters found in
the original painting. Painting’s surrounding space (a bright room having some
furniture) has been digitally reconstructed in a VR environment, taking into
consideration the limited resources of mobile devices.
ARTISTS prior version was a mobile application in which users were also able to
interact with the 3D version of a painting by just tapping on mobile device’s screen,
thus without totally immersion to the VR environment. Application settings like
sound, running scenarios, animations etc were depending on user’s profile and
interests, a functionality that still stands in ARTISTS, but with the use of more
accurate methodologies.
Technologies used in ARTISTS
Context Awareness
In ARTISTS design, we take into consideration parts of the context like the ambient
noise level, processing power of the mobile device and screen resolution, trying to
improve users’ experience regardless of environmental conditions. In particular, in a
quite noisy environment (to the noise level of 50dB), sound volume can be increased
up to 50%, whilst in extremely noisy conditions (noise level more than 70dB),
application audio volume mutes to avoid Lombard effect (Varadarajan Vaishnevi,
Hansen John H.L., 2006). In a full scale application of ARTISTS, noise levels would
be measured by a sensors network, in accordance with user’s position in space.
Furthermore, processing power of the portable device in use can be a crucial asset
which can deeply affect user experience. Ιnsufficient resources could affect the
reproduction of high-resolution 3D animation and graphics needed to construct the
VR environment, while also screen resolution could be a negative factor in displaying
high resolution graphics. A short benchmark on the background, during application
installation can easily adjust applications’ settings to the appropriate level based in
devices’ capabilities before the initialization of the application, thus avoiding
malfunctions during users’ experience.
Personalized User Experience
In our case, we use the User Personas method, which categorize users based on their
profile during a museum visit. User Personas (Morris, Hargreaves and McIntyre,
2004) are not real people but avatars created studying real people’s characteristics.
We use 4 User Personas with the names “Follower”, “Browser”, “Searcher” and
“Researcher”. Followers try to follow any guidance provided by the museum or
cultural site, trying also to learn something by it. Browsers won’t follow a guide but
go anywhere, in every place that looks interesting, and then, they search for
information about it. Searchers will search and collect detailed information on specific
exhibits or collections whilst Researchers step further on a scientific research about
specific exhibits (Konstantakis et al., 2018).
Gesture Recognition and 3D Interaction
Gesture recognition refers to computers’ ability to understand gestures involving
physical movements of multiple body parts (fingers, arms, hands, head, feet, etc) and
execute commands based on the corresponding gesture, thus allowing interaction with
the computer environment. Many gesture recognition approaches suggest that gestures
used as interaction methods between humans, can also been successfully applied as a
natural and intuitive way to interact with machines [Ren et al., 2016][Yeo et al, 2015].
In ARTISTS framework, we use the Leap Motion controller to track users’ hands and
match their movements with commands in the virtual environment. As users’ mobile
device is found into a Google Cardboard type VR device, it is impossible to tap on the
screen. Leap Motion API gives us the tools to interact with the app interface by using
hands. Simple tasks like selecting a character, dragging the volume slider, selecting
from menus and pressing on UI buttons can be done with natural hand movements in
space, in a quite accurate, intuitive and entertaining way.
User Personas
The design of personas as ‘fictional’ characters is considered as a very consistent and
representative way to define actual users and their goals. However, it is important to
clarify the exact number of personas in each occasion in order to focus on the visitor
profiles to be examined. On ARTISTS, we take into consideration these UPs and their
characteristics and we create more Personas by splitting Followers and Browsers into
3 Levels. Searchers and Researchers are combined and split into 2 Levels. These
Levels have a quantitative meaning. For example, Level 2 Researcher has done more
research and shows more of the initial Researcher characteristics than Level 1
Researcher.
In order to match each museum (or any other cultural site) visitor to an ARTIST
persona, the system collects and process various data about visitors. Data mining is
ARTISTS involves no user interference or preparation and it’s a 3-stages process:
1. Face recognition: Using Microsoft Cognitive Services, user age and emotions
are calculated by their face picture taken from the device’s front camera that is
sent over network. In addition, a database of visitors is created, turning every
possible upcoming visit into a more successfully personalized experience.
2. Social networks data mining: Using data mining algorithms, visitor’s data
(profile and prior experience) are extracted from user social profiles
(Facebook, Twitter or Instagram). Fully compatible with GDPR rules,
algorithms can only use data that users expose as public.
3. Behavior study: Sensors embedded into the visiting area monitor visitors’ path
and behavior into space, providing ARTISTS more personalization data.
System Architecture
Image 1: System architecture in ARTISTS
ARTISTS is a Client – Server system, as shown in Image 1. Core of the system is a
server, located either in a museum (or any cultural site) or in a remote position. Server
supports communication between database, application and sensors network (installed
in museum). Furthermore, more server tasks are responsible for matching visitors to
predefined personas, or displaying multimedia for the VR environment.
The mobile application creates the appropriate interface between user and ARTISTS
system. Depending on visitors’ profile, the system shows a different scenario and
service. Server also is responsible for handling sensors’ and Smart Objects (SO) input
that can alter applications’ content.
Use Case Scenario
After getting necessary visitor data and assigning one persona from Table 1, one of
the 19 usage scenarios may initiate. These scenarios are 19 in total and matching a
visitor to a scenario is a dynamic process. For example, user can start visiting a
museum as a Level 3 Follower, but after a while, his behavior can turn him into Level
1 Browser and then Level 2 Browser. This happens because behavior monitoring is an
ongoing process that gives feedback data which can eventually change the flow of
user experience. Each one of the scenarios in Table 2 is different in functionality,
interactivity, display quality and load, audio (Table 2).
Conclusion - Future work
In this work, we describe the ARTISTS framework, a mobile application that displays
a VR reconstructed environment of a painting, and immerses users allowing them to
Table 1: Interaction – usage scenarios in ARTISTS.
Image 2: The VR representation famous painting “Children Concert” created by Georgios Iakovidis
interact with its 3D aspects. We used the Leap Motion controller as a sensor for
detecting gestures, alongside with Unity, Microsoft’s Azure Cognitive Services and
Android Studio for the implementation of the application and the MySQL database
that stores the 3D environment and painting’s data.
Our next step includes the ARTISTS evaluation stage, in which we will test our
framework to evaluate user’s experience and the efficiency of our integrated
technologies.
References
Antoniou, A. & Lepouras, G. (2010). Modelling visitors' profiles: A study to investigate
adaptation aspects for museum learning technologies. J. Comput. Cult. Herit. 3 (2), Article
No.7, pp. 1-19.
Antoniou Angeliki et al. (2016). Capturing the Visitor Profile for a Personalized Mobile
Museum Experience: an Indirect Approach, University of Peloponnese, University of Athens,
Pompeu Fabra University, CEUR Workshop Proceedings, Vol-1618.
Chang Kuo-En et al. (2014). Development and behavioral pattern analysis of a mobile guide
system with augmented reality for painting appreciation instruction in an art museum,
Elsevier Computers & Education 71, p. 185-197.
Dey A., Abowd G., Salber D. (2001). A conceptual framework and toolkit for supporting the
rapid prototyping of context-aware applications in special issue on context-aware computing,
Human Computer Interaction, J. 16 (2-4), pp. 97-166.
Eardley W.A. et al. (2016). An Ontology Engineering Approach to User Profiling for Virtual
Tours of Museums and Galleries, International Journal of Knowledge Engineering, Vol. 2.
Katz Shahar et al. (2014). Preparing Personalized Multimedia Presentations for a Mobile
Museum Visitors’ Guide – a Methodological Approach, The University of Haifa - Israel, ITC-
irst – Italy.
Konstantakis Markos et al. (2017). Formalising and evaluating Cultural User Experience,
University of the Aegean, IEEE.
Konstantakis Markos et al. (2018). A Methodology for Optimised Cultural User peRsonas
Experience - CURE Architecture, British HCI 2018 Conference, Belfast, Northern Ireland,
2018.
Morris G. et al. (2004). Learning Journeys: Using technology to connect the four stages of
meaning making, Birmingham: Morris, Hargreaves, McIntyre Website.
Naismith Laura, Smith M. Paul (2006). Using mobile technologies for multimedia tours in a
traditional museum setting, mLearn 2006: Across generations and cultures, p.23, Canada.
Pujol Laia et al. (2012). Personalizing interactive digital storytelling in archaeological
museums: the CHESS project, The CHESS Consortium.
Ren, Z., Yuan, J., Meng, J., & Zhang, Z. (2016). Robust part-based hand gesture recognition
using kinect sensor. IEEE Transactions on Multimedia, 15.
Roto V. et al. (2010). User Experience white paper. Bringing clarity to the concept of user
experience, Dagstuhl Seminar on Demarcating User Experience.
Varadarajan Vaishnevi S., Hansen John H.L. (2006). Analysis of Lombard effect under
different types and levels of noise with application to In-set Speaker ID systems, University of
Texas at Dallas, USA.
Yeo, H. S., Lee, B. G., & Lim, H. (2015). Hand tracking and gesture recognition system for
human-computer interaction using low-cost hardware. Multimedia Tools and Applications,
74(8), 2687-2715.