Content uploaded by Human Esmaeili
Author content
All content in this area was uploaded by Human Esmaeili on Oct 10, 2017
Content may be subject to copyright.
PepperGram With Interactive Control
Chi How Fong
Universiti of Tunku Abdul Rahman
Selangor, Malaysia
chihowsg93@gmail.com, chihow@1utar.my
Mark Billinghurst
University of South Australia
Adelaide, Australia.
mark.billinghurst@unisa.edu.au
Zi Siang See
Sunway University
Selangor, Malaysia
zisiangsee@sunway.edu.my
Human Esmaeili
Sunway University
Selangor, Malaysia
hesmaeili@sunway.edu.my
AbstractA peppergram allows a user to experience a visual
image floating in the air. This paper explores how to add
interactive controls to a peppergram. An investigation was
conducted to explore the suitability of two different integrative
methods for navigating a peppergram as a multimedia
presentation tool; a Myo armband, and touch screen input. These
methods were used with prototype 3D visual images experienced
on a high-resolution mobile tablet device. A pilot study was
conducted to evaluate the user experience. We found that using
freehand gesture input with the Myo could be one way to provide
interaction with peppergram virtual content without requiring
any touch input. These results could be used as the basis for
further development of interactive peppergram displays.
KeywordsPeppergram, Myo Armband, Virtual Remote
Control, Interaction, Peppers Ghost, and Augmented Display.
I. INTRODUCTION
Peppers Ghost was invented in 1862 by John Pepper [1] to
wow unsuspecting theatre going audience members. It is an
illusion that creates the impression that a ghost has appeared on
stage next to the actors. This is accomplished by placing a piece
of glass at an angle placed between the audience and the stage,
through which light is shone, reflected from an actor below the
stage, creating the optical illusion (see Fig. 1). This approach is
commonly known as a peppergram in the creative and
information technology industries [2]. Peppergram are still used
today, such as for interactive marketing at Berjaya Times
Square, Kuala Lumpur and showing Dita Von Teese appearing
in the Times Square Hologram, at Studio City Macau [3].
Fig. 1. Peppers Ghost as shown in 1862.
Most peppergrams are non-interactive, just playing back a
pre-recorded video of a performer. The main contribution of this
study is to explore different ways to interact with peppergram
content. In particular, we use the Myo armband, a device that
measures electrical activity in forearm muscles, and so can
detect different arm movements and hand gestures [4]. In our
work we use the Myo to translate hand gestures into interactive
control of a peppergram application. We also compare this to
more traditional touch screen input to determine which method
may provide a preferred visual experience. This experiment is
part of broader research into how to add interactivity to
peppergram displays.
In the rest of the paper we first describe background research
on interactive peppergrams, then discuss how we implemented
our system and a user study conducted with system. Finally, we
finish with a discussion and conclusion with directions for future
research.
II. BACKGROUND RESEARCH
Since 1862 peppergrams have been used in many settings
and different form factors, from museum installations [5] to
conference stages. Most recently, the emergence of smart
phones, and tablets has enabled peppergrams to be delivered in
mobile settings using small foldable transparent screen.
Traditionally peppergram are non-interactive, just showing
prerecorded experience to the viewer. However, recently a
number of companies and researchers have explored how to add
interactivity. For example, PRHolo [6] have combined a 3D
depth sensor with a peppergram installation to support natural
gesture interaction with users. Another study has explored using
an infrared sensor and Microsoft Kinect interaction with a
peppergram [7]. Similarly, in the Calderan project [8] a Leap
Motion Controller was used to enabled people to reach out and
translate and rotate the virtual images. Thange et al. [9] show
how a simple motion sensor can be used to add interactivity to a
peppergram.
Real objects can also be used to support interaction with
peppergrams. For example, computer vision has been used to
track real cards onto which the peppergram virtual images
appear [10]. Moving the cards enables the user to see the virtual
content from different viewpoints. Other people have explored
how to use external devices, such as smart phone to support
touch screen input, mapping 2D input to 3D object manipulation.
[11].
These systems either use embedded sensors in them or
require the user to carry a handheld device, and have some
limitations. For example, the viewer has to put their hands into
the Leap Motion interaction volume to capture the user input. To
overcome these limitations, we are interested in exploring how
to use body worn devices for input, and in particular the Myo
armband. In the next section we describe the peppergram
prototype we developed using the Myo armband.
III. PROPOSED APPROACH
The peppergram prototype we developed was made up of
stiff, reflective plastic bent into a pyramid shape and placed on
a smartphone or tablet, with each of the four sides are at a 45
angle to the screen. As shown in Fig. 2 the pyramid was 3 cm x
9 cm x 11 cm in size and Figure 5 shows an image generated in
it pyramid when it was placed on a tablet.
Fig. 2. Dimensions of the peppergram pyramid.
Fig. 3 shows how the peppergram on the tablet worked.
Light from graphics shown on the phone screen bounces off
the plastic pyramid and is reflected to the viewers eyes. The
plastic is transparent so the viewer can also see through the
peppergram, so the virtual model shown appears to float in
space within the pyramid
Fig. 3. Viewing an image inside the peppergram.
The tablet device used for our study was a Microsoft Surface
Pro 4 with a 12.3 inch display and a resolution of 2736 x 1823
pixels (267 PPI). On the Microsoft Surface Pro we showed
videos of 3D objects, edited and placed in four different
rotations, one for each face of the peppergram. Fig. 4 shows a
screenshot of Adobe Premier Pro application, creating a four-
face video used for prototype demonstration, and Fig. 5 shows
the final view of the 3D model in the peppergram.
Fig. 4. A snapshot of the four phase video editing settings.
Fig. 5. Peppergram image on the Surface Pro tablet.
In order to add interaction to the peppergram, we used the
Myo armband as a gesture controller to remote control the video
played in the VLC Media Player. Fig. 6 shows the Myo
Armband and Fig. 7 the gestures for controlling the armband and
mapped onto video controls. In this case the Finger spread
gesture is assigned to the Play/Pause function, Fist to control
the volume and Rotate to adjust the volume up and down,
Wave Left, to rewind the video, Wave Right to fast-forward
the video, and finally the, Double Tap gesture was mapped to
the timed unlock. As a result, this provides users with some level
of control over for the peppergram illusion.
The Myo armband works by using proprietary EMG sensors,
which are built into on the armband. The Myo armband
measures electrical activity from muscles to detect five hand
gestures [12]. Using a 9-axis IMU (Inertial Measurement Unit),
it also senses the forearm motion, orientation and rotation.
Developers can combine the pre-set gestures with arm motions
to create new gestures. Developers are also able to access the
raw EMG data from the Myo to create their own custom
gestures. A custom calibration profile tool in the Armband
Manager of Myo Connect allows users to record their own
motions that map to the four EMG-based gestures. The Myo
armband transmits gesture information over a Bluetooth Smart
Connection to communicate with compatible devices [13].
The Myo can be used up to 15m (50ft) meters away from the
peppergram, so there are several advantages in using Myo for
interactive control of the peppergram display. The wireless
capability makes it easy to operate and users are able to gain
control at a distance away from peppergram. This mean that it is
especially suitable for a group presentation where viewers can
experience the visual effect of peppergram while a presenter
controls the floating object without physically touching the
display surface or being in the way of the viewers. Moreover,
the Myo and peppergram are portable and hence can be used
anywhere.
Fig. 6. Myo Armband.
Fig. 7. Armband control with options of gesture input.
In addition to the Myo input we also used a simple touch
screen controller to interact with the virtual peppergram video
content. The use of a touch screen input requires the user to
touch their finger onto the display screen of the tablets. In this
case, the user can touch on menu bar buttons of the VLC video
player application to control the interactivity of the content, such
as the play/pause button, rewind, etc.
IV. USER STUDY
A pilot study was conducted to evaluate the Myo technology
for gesture interaction with the peppergram and compare it to the
more traditional touch input. A set of 20 participants were
involved in the user study, 16 males, and 4 female ranging in age
between 18 to 23 years old, 24 to 29 years old, and 30 to 35 years
old, (SD=3.30). The focus of the user study was to measure the
usefulness of the interactive control capability with the
peppergram display. Participants were requested to wear a Myo
armband device on the arm, and look 90 degrees towards
peppergram pyramid to see the illusion, and perform some hand
gesture to control the movement of illusion.
The experiment had two conditions: C1 Touchscreen: using
touchscreen gesture controls, and C2 Myo: using Myo armband
gesture controls. Each participant experienced both conditions
in a counterbalanced order.
In the Touchscreen condition (C1), each participant was
briefly taught how the touchscreen worked on the tablet by
touching functional buttons in the application. Participants also
tried perform scrolling to the left and to right of the taskbar in
VLC Media Player application.
In the Myo condition (C2), before performing any hand
gesture, each participant was briefly taught about the Myo
armband and peppergram history and theory. Fig. 8 shows a
participants testing the Myo armband and peppergram. An
experimenter explained the process and provided a complete
demonstration of the Myo Armband hand gestures.
Fig. 8. Participants testing the Myo Armband and PepperGram.
In each condition, participants were asked to look at the
peppergram and perform interaction with Myo hand gestures, or
touch screen input. After each condition, we collected
participants feedback on how easy it was to use the control
technology (either Myo or touchscreen input). This was done by
collecting qualitative feedback responses to the questions shown
in table 1. Answers where captured on a Likert scale of 1 to 7 in
which 1 was Strongly Disagree and 7 was Strongly Agree.
After both conditions we interviewed participants for more
feedback about the controls. Our key interest was to understand
the perceived ease-of-use and usefulness of the armband by the
participants, and how they described their experience with the
peppergram.
Table 1: Survey Questions
Q1 I found it easy to use
Q2 I found it natural to use
Q3 I found it reliable
Q4 I found it physically challenging
Q5 I found it mentally challenging
Q6 I found it useful
Fig. 9 shows the average results of the C1 and C2 survey
questions. Overall there was no difference in the responses to the
survey questions between the Myo and touchscreen input in
terms of perceived ease of use (Q1) and usefulness (Q6), but
there was in the questions relating to physical (Q4) and mental
(Q5) challenge, how natural it was to use (Q2), how reliable it
was (Q3), and how challenging (Q4).
A Wilcoxon Signed Rank test was used to analyze the results
to check for significant difference between the results of the
using the touchscreen gesture control (C1) and the Myo armband
gesture control (C2). For Q1, using a one-tailed test we found
that participants felt it easy to use the touchscreen gesture control
with no significant difference between conditions, Z = - 1.223,
p = . For Q2, finding the gesture natural to use, there was
significant difference between C1, and C2, with Z = - 2.103, p =
. There was also a significant difference between
conditions it terms of how reliable participants felt each
condition was (Q3), Z = - 1.712, p = In terms of the
physical challenge (Q4), participants felt that the Myo armband
(C2) was significantly more challenging (Q4) than the
touchscreen (C1), Z = - 3.059, p = 0.001. Similarly, C2 was felt
to be more mentally challenging (Q5) than C1, Z = - 2.48, p =
0.007. Finally, there was no difference between the conditions
in terms of usefulness. (Q6), Z = - 1.192, p = 0.117. Overall,
these results show that the touchscreen gesture interface was
better than the Myo gesture input.
Fig. 9. Average results of C1 and C2 survey questions.
We also asked participants the following questions about the
peppergram experience, using a Likert scale rating from 1 to 7
(1 = Strongly Disagree, 7 = Strongly Agree):
QA How much experience do you have with Peppergram?
QB Do you feel the level of details or display resolution is
important to you?
QC Do you feel the users ability to interact with display
content is important?
Fig. 10 shows the average results. Only a few participants
had experience with peppergrams (QA). More than half of the
participants agreed that the level of details or display resolution
was important (QB), and most strongly agreed that the ability to
interact with the displayed content was important (QC).
Fig. 10. Average results of QA, QB and QC survey questions.
We also asked participants for their comments or suggestions
on the peppergram and controls. Some users said that they found
the peppergram an interesting technology, with comments such
as ... interesting, I have never seen a peppergram before....
However, they felt the peppergram could be improved in number
of ways, such as . peppergram can be improved, by creating
a bigger virtual illusion..., They also felt that there could be
enhancements to the gesture controls used, with comments such
as more features of armband could be more utilizde such as
zoom in, and zoom out of the illusion/video prototype
We observed that the participants who were are new to the
armband found it challenging to control and repeat the hand
gestures. Another difficulty experienced by the users is that Myo
becomes locked once a gesture is performed, and users need to
perform an unlock gesture in order to perform another input
gesture. Many inexperienced users ended up being locked and
spent a lot of time to unlocking the Myo without successfully
controlling content shown in peppergram. This situation may be
overcome with practice, enabling the user to gain confidence in
operating the armband input for the peppergram.
V. DISCUSSION
Most of the experiment participants felt that they were
extremely familiar with the touchscreen input as they all had
personal mobile devices with touchscreens. However, the Myo
armband gesture control was new to all participants. Despite
this, participants felt that there was no difference in perceived
ease of use and usefulness between the touch input and Myo
gesture input. This is a positive result that shows the devices like
the Myo could potentially be used for interactive control for
peppergram experiences.
However, most users felt that using the Myo armband was
more physically challenging compared to the touchscreen input.
One of the reasons for this is that different users produce
different nerve muscle signals and so each user should carefully
calibrate the Myo before using it. Different people also have
different body size types that might affect the reading time
recognition delay. For example, thin people often have small
forearm muscles. Using the Myo also requires performing a
range of different hand gestures that require more physical input
than simply touching a screen.
However, in the future the Myo armband could be reliable
and natural to use and reliable. For example, the armband could
be improved for better hand gesture recognition. Through this
research, interactive control could be more improved, and it
could be more easy to correctly recognize the hand gestures.
VI. CONCLUSION
In this study, we added interaction control to a peppergram
using a Myo armband control. The use of the Myo armband was
compared to more traditional touchscreen input. It was observed
that participating users were comfortable close to the touch
screen. Participants also made several good suggestions for
improving the technologies such as better gesture recognition,
and reducing the recognition timing delay.
The perceived ease-of-use and usefulness of the armband
control was found to be no different to the familiar touchscreen
input when using the peppergram. Participants described what
they thought the advantages were of armband and touchscreen
input, and they also mentioned areas for further improvement.
For example, accessing the EMG readings, gives the possibility
to further extend the set of supported hand gestures by creating
a self-written hand gesture. Using this it might be worth
considering a different type of unlock pattern for the timed lock
in the Myo armband.
In the future, we would like to further improve our efforts in
designing the peppergram content that works with different
degrees of controls and explore improved approaches of using
interactive technologies. The visual attention required, as well as
the social acceptance of possible interactive control technologies
should be examined, especially in contrast to other technologies
such as speech recognition, and eye gaze input.
5.15
2.9
3.7
4.55
4.45
4.75
5.5
2.15
2.5
5.1
5.2
5.15
0 1 2 3 4 5 6 7
Q6
Q5
Q4
Q3
Q2
Q1
PepperGram With Interactive Control User Feedback
Category 1 : Peppergram with Touchscreen Gesture Control
Category 2: Peppergram with Myo Armband G esture Control
6.25
3.85
2.65
0 1 2 3 4 5 6 7
QC
QB
QA
Likert Scale of Question A, B and C
REFERENCES
[1] R. Sarah, Figthing Ghosts, Playing Whist, and Fencing with Fire: Three
Technologies of Illusion in Perfomance in Nineteeth-Century London,
University of Toronto, 2012, pp. 57.
[2] OSA, Activity Guide PeppersGhost, Laser Classroom, Retrieved 14
May, 2016.
[3] Trip Advisor. Studio City Macau, Accessed Oct 31st 2016.
https://en.tripadvisor.com.hk/LocationPhotoDirectLink-g664891-
d8331360-i192229230-Studio_City_Macau-Macau.html
[4] Myo TM. Official Website. Accesseed Oct 19th 2016.
https://www.myo.com/
[5] Shane Warne-Cricket Found Me. National Sports Museum, Melbourne,
Australia, Accessed Oct 20th 2016.
http://www.nsm.org.au/Exhibitions/Shane Warne Hologram.aspx
[6] A. Ricardo, PRHOLO: Interactive Holographic Public Relations,
Proceeding of the Third International Conference Advances in
Computing, Communcation and Information Techonology, 2015.
[7] E.S. Malinverni, E. dAnnibale, E. Frontoni, A. Mancini and C. A. Bozzi,
Multimedia Discovery of The Leonardos Vitruvian Man, SCIRES-IT-
Scientific Research and Information Technology. vol. 5(1), pp. 69-76,
2015.
[8] Leap Motion Blog, From Idea to Illusion: Creating Calderan, Accessed
Oct 16th 2016.
http://blog.leapmotion.com/from-idea-to-illusion-creating-calderan
[9] T. Reshma, S. Prachi, K. Vinayak and V. Jain, Interactive Holograms
using Pepper Ghost Pyramid, International Journal for Scientific
Research & Development (IJSRD), vol. 4(1), 2016.
[10] Makers.htxt.africa, Hack yourself togther some pokemon holograms,
Accessed Oct 16th 2016.
http://www.htxt.co.za/2016/02/22/90701/
[11] Conran Holo TM, Official Website, Accessed Oct 16th 2016.
http://beagleholo.com/
[12] Myo Support, How does the armband work, Accessed Oct 19th 2016.
https://support.getmyo.com/hc/en-us/articles/202532376-How-
does-the-Myo-armband-work
[13] Myo TM. Official Website. Accesseed Oct 19th 2016.
https://www.myo.com/
