Content uploaded by Jari Kangas
Author content
All content in this area was uploaded by Jari Kangas on Jun 18, 2014
Content may be subject to copyright.
Gaze Gestures and Haptic Feedback in Mobile Devices
Jari Kangas, Deepak Akkil, Jussi Rantala, Poika Isokoski,
Päivi Majaranta and Roope Raisamo
Tampere Unit for Computer-Human Interaction, School of Information Sciences
University of Tampere, Finland
{jari.kangas, deepak.akkil, jussi.e.rantala, poika.isokoski,
paivi.majaranta, roope.raisamo}@uta.fi
ABSTRACT
Anticipating the emergence of gaze tracking capable mobile
devices, we are investigating the use of gaze as an input
modality in handheld mobile devices. We conducted a
study of combining gaze gestures with vibrotactile
feedback. Gaze gestures were used as an input method in a
mobile device and vibrotactile feedback as a new
alternative way to give confirmation of interaction events.
Our results show that vibrotactile feedback significantly
improved the use of gaze gestures. The tasks were
completed faster and rated easier and more comfortable
when vibrotactile feedback was provided.
Author Keywords
Gaze tracking; gaze interaction; haptic feedback.
ACM Classification Keywords
H.5.2. User interfaces: Input devices and strategies.
INTRODUCTION
Availability of low-cost miniature video cameras and low-
power computing hardware is now making it possible to
build affordable mobile devices such as smartphones or
computing tablets with gaze tracking capability. Also,
prototypes of affordable head mounted gaze trackers are
being built (e.g. [10]), which can be used together with
mobile devices. The conventional way of interacting by eye
gaze is to fixate the gaze to an on-screen object for a pre-
defined dwell time. However, this is not optimal in mobile
contexts because objects on small displays tend to be small
and the handheld device is moving slightly [4].
In some cases the gaze accuracy problems can be corrected
by using other modalities, like touch input [14, 15]. Gaze
gestures provide an alternative method of gaze interaction.
They are known to be more tolerant to small calibration
shifts, display and tracker movement, and also suitable for
interaction with small display objects [2, 3, 4, 7]. The use of
gaze gestures has been studied, for example, in gaming [9],
text typing [16], control [12] and drawing [5].
Even though gaze gestures have been shown to be a
potential interaction method, one existing challenge is to
provide sufficient feedback of the interaction. Dybdal et al.
[4] found that user’s cognitive load is higher in gesture
based interaction than in dwell time based interaction. They
proposed that the interface design should be improved, e.g.,
by adding audio feedback of gestures. In general,
appropriate real-time feedback of user’s actions is known to
reduce mental load and make the interaction easy and fast
[11]. In the context of gaze gestures, instantaneous
feedback would help in confirming that the input has been
recognized and in coping with errors such as incorrectly
recognized eye strokes. An early feedback already during a
gesture could also be beneficial for correct completion of
the gesture [12].
While the use of visual and audio feedback is possible when
using gaze gestures, they both have limitations. Visual
feedback on a mobile display can be difficult to perceive
during eye movements, and may be too late if given after
the gesture completion. Also, auditory feedback is not
suitable for private interaction or in noisy environments [1].
To date, little attention has been paid to studying the use of
haptic feedback with gaze input. This could possibly be due
to the fact that providing haptic stimulation to the user has
required separate feedback devices. However, in the context
of mobile interaction, the input and output capabilities are
combined in a single device. Most current mobile devices
are equipped with vibrotactile actuators. These actuators
have been shown to be useful, for example, in improving
both the performance and subjective experience of virtual
keyboard use on touchscreen devices [1, 6]. Combining
vibrotactile feedback with gaze gestures could allow for
novel types of interaction.
Motivated by this, our aim was to find out whether the use
of vibrotactile feedback is beneficial when using gaze
gestures to control a mobile device. In this paper, we
present a study on comparing different types of vibrotactile
feedback while performing list-based tasks using gaze
gestures. List task was chosen as it is familiar to users, fast
to do and requires a number of different simple commands.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for
components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to
post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from Permissions@acm.org.
CHI 2014, April 26 - May 01 2014, Toronto, ON, Canada
Copyright 2014 ACM 978-1-4503-2473-1/14/04…$15.00.
http://dx.doi.org/10.1145/2556288.2557040
We selected a simplified setup to get basic results of the
utility of vibrotactile feedback. Task completion times and
subjective experiences were measured to investigate
whether the addition of vibrotactile feedback improves the
interaction. As far as we know, the combination of gaze
input and haptic output has not been explored previously.
EXPERIMENT
The experiment was designed to resemble a scenario where
a person operates a handheld mobile phone through gaze
gestures. The person is not able to use other input
modalities. For test purpose we implemented an application
that simulated a contact list in a mobile phone. The main
objective was to study the effects of vibrotactile feedback
on efficiency and subjective ratings. The secondary
objective was to study the user’s overall perception of gaze
gestures as an input method for a mobile device.
Participants
We recruited 12 volunteer participants (2 females and 10
males), aged between 16 and 50 years from the University
community. All participants were familiar with mobile
technology, and ten were familiar with haptic feedback. Ten
participants were also familiar with gaze tracking
technology. Two of the participants had corrected vision.
Apparatus
We used a Tobii T60 gaze tracker and Nokia Lumia 900
mobile device to simulate an eyetracking capable mobile
phone. Once gaze input data was collected by the Tobii
device, gaze gestures were recognized in a PC and
transferred to the phone through a USB based socket
connection. The control logic for the application ran on the
mobile device based on asynchronous gaze gesture events
sent from the PC. The vibrotactile feedback was generated
by the phone’s built-in vibration motor.
Gaze Gestures
We used two stroke gaze gestures, and utilized off-screen
gazing [8] to make the input area larger than the display.
In the beginning of a gesture the user was always looking at
the display of the phone. The first stroke of a gesture moved
over the edge of the device, and the second stroke returned
back to the device. In order to differentiate between normal
eye movement and gaze gesture commands, we defined a
maximum time of 500 ms (long enough for a fixation based
on pilot testing) for the duration of the fixation outside of
the device. Gestures lasting longer were not recognized.
The following gestures were used:
Up crossed the top edge of the device and moved
the selection one position upwards in the list
Down crossed the bottom edge of the device and
moved the selection one position downwards.
Select crossed the right edge of the device and
activated the presently selected name.
Cancel crossed the left edge of the device and
returned to the list.
Haptic Feedback
The system recognized the first stroke and the full gesture
separately making it possible to give feedback half-way
through the gestures. The length of the vibrotactile
feedback was set to 20 milliseconds that we found to be
long enough to be felt by all participants in pilot testing.
To study what type of feedback would be the most efficient,
we defined four haptic feedback conditions (see Figure 1):
No: No haptic feedback.
Out: Haptic feedback given when a stroke,
originating from inside the device to outside the
device, was recognized.
Full: Haptic feedback given when the second
stroke, originating from outside the device to
inside the device, was recognized.
Both: Haptic feedback combining Out and Full.
Procedure
In the beginning of the experiment each participant filled in
a pre-experiment questionnaire. This was followed by an
introduction to the experiment and calibration of the gaze
tracker. Then the participant was instructed to hold the
mobile device in his/her hand so that the haptic feedback
could be felt. The back of the holding hand touched a small
piece of foam mounted on top of the eye tracker’s display.
This allowed small movements of the device, but reduced
the likelihood of the phone drifting away from the intended
position. The participant’s hand was also supported from
elbow to prevent fatigue during the experiment. See Figure
2 for the arrangement.
The task was to find a specific name in a list, select the
name and make a simulated call. The participant saw a list
of 18 names (part of which is shown in Figure 2). When the
Figure 1. Haptic feedback conditions for the Up gesture.
Figure 2. The participant held the device in front of the
tracker (left). An example list of names visible at the
mobile’s display (right).
application was started the list focus was on the topmost
entry. The list was ordered alphabetically and long enough
not to fit on the display. After a successful call the system
would pause for five seconds before automatically returning
to the list of names. During this pause, the participant was
given the next name to find. The search for subsequent
names started with focus on the previously called name.
A block of four calls was completed in each feedback
condition. In addition each block contained a short practice
session where the participants could ensure that the system
worked and they could feel the haptic feedback. After the
four calls the participants evaluated the easiness and
comfortableness of using the particular feedback condition
by filling in rating scales ranging from 1 (very
uncomfortable / very difficult) to 7 (very comfortable / very
easy). All the participants used the same name list and
made calls to the same names to eliminate variability due to
the position of the names on the list. The names were
different in different feedback conditions. However, they
were chosen so that the same number of gestures was
required to complete each block. The order of the feedback
conditions was counterbalanced between participants using
a Latin square. The test design was within subjects.
As we expected a noticeable learning effect in the use, we
conducted two identical sessions in the experiment. That is,
all four feedback conditions were performed twice. The
first session was intended for training the participants. After
completing both sessions, a final questionnaire was used to
determine which of the four feedback conditions the
participants felt was the easiest / most comfortable to use.
Results
Task completion times
The median block completion times in seconds for all eight
blocks are shown in Table 1 in the order of presentation
(from T1 to T8) to show the effect of learning. The
feedback conditions varied in each block according to our
counterbalancing scheme. The block completion times were
longest in the first two blocks.
In order to eliminate the learning effect, only data from the
second session (T5-T8) was used in further analysis. The
block completion times for the four different feedback
conditions are shown in Figure 3. The largest difference
(17%) in the completion times was between the median
time for condition No and the median time for condition
Out.
Our data distributions did not meet the normality
assumption of ANOVA. Because of this, we used a simple
permutation test to estimate the significance of the observed
pairwise completion time differences between the
conditions in the sample. Assuming a null hypothesis of no
difference we computed 10,000 resamples with random
reversals of the differences between the conditions. Then
we observed how often the median difference was more
extreme than the observed value. The results showed that
completion times for condition Out were significantly lower
than for conditions No and Full (p<=0.05 in both cases).
The differences between other feedback conditions were
not statistically significant.
Gestures per Action
In most of the test cases the participants made unnecessary
gestures to complete the task. We defined a Gestures per
Action (GPA) measure as the ratio of the number of
performed gestures to the minimum number of gestures
needed. The value of GPA increases if the user does wrong
selections, overshoots the focus and needs further gestures
to correct these errors.
Table 2 shows the average GPA for the different conditions.
The participants performed more gestures to complete the
task in condition No than in any other condition. The
biggest difference is between conditions No and Out, where
about 15% more gestures were used to complete the task in
condition No.
Subjective evaluations
The results of subjective evaluations gathered after each
condition showed that condition No was rated as
significantly (permutation test as before) more
uncomfortable and difficult to use compared to the other
conditions (Figure 4).
Also, in the post-experimental questionnaire, six out of
twelve participants felt that condition Both was overall the
most comfortable to use. In terms of easiness, eight
participants preferred condition Both. Notably, none of the
participants preferred condition No after the experiment.
Condition
No
Full
Out
Both
GPA
1.33
1.18
1.16
1.19
Table 2. Gestures per action for different feedback conditions.
Slot
T1
T2
T3
T4
T5
T6
T7
T8
Time
69.2
47.5
44.1
43.2
44.6
45.0
45.1
44.3
Table 1. The median block completion times in seconds in
different time slots of the experiment.
Figure 3. The block completion times in seconds for different
feedback conditions.
Ten participants said that their eyes were more tired after
the experiment. Eight participants would use gaze gestures
in a mobile device if that was available. When asked about
possible use situations, most listed special situations where
the hands were not available.
DISCUSSION AND CONCLUSIONS
Our results showed that the use of haptic feedback
increased both the efficiency and subjective experiences of
gaze-based interaction. Out of the three different haptic
feedback conditions, the most efficient ones in terms of task
completion times were Out and Both that provided feedback
when the gaze was outside of the display borders. This
indicates that the participants could make use of the
feedback while doing the gestures. The Gestures per Action
measure suggested that the addition of feedback helped
because it allowed carrying out the tasks with fewer errors.
Conditions Out and Both that contained the haptic feedback
when gaze was outside the device were faster than the other
conditions. Participants also subjectively preferred the
haptic feedback conditions over the non-haptic condition.
There was a significant learning effect during the first one
or two blocks, but after that block completion times leveled
off. However, overall the experiment was very short and
further learning might happen in extended use.
Overall we can conclude that haptic feedback on gaze
events can improve user performance and satisfaction on
handheld devices especially in cases where visual or audio
feedback is difficult to arrange. Our findings could be
utilized in any gaze-based mobile applications, for example,
for giving simple commands without touch input. Further
work investigating the combination of gaze input and haptic
output should be undertaken to understand the benefits
more widely. Gaze gestures combined with haptic feedback
provide an easy input method when hands are busy. Further
work is also needed to study the effect in combinations with
other input and output modalities.
ACKNOWLEDGMENTS
We thank the members of TAUCHI who provided helpful
comments on different versions of this paper. This work
was funded by the Academy of Finland, projects Haptic
Gaze Interaction (decisions numbers 260026 and 260179)
and Mind Picture Image (decision 266285).
REFERENCES
1. Brewster, S., Chohan, F., and Brown, L. Tactile
Feedback for Mobile Interactions. In CHI’07, 159-162,
ACM Press, 2007.
2. Drewes, H., De Luca, A., and Schmidt, A. Eye-Gaze
Interaction for Mobile Phones. In Mobility’07, 364-371,
ACM Press, New York, 2007.
3. Drewes, H., and Schmidt, A. Interacting with the
Computer using Gaze Gestures. In Proc. INTERACT
2007, 475–488. Springer, 2007.
4. Dybdal, M.L., San Agustin, J., and Hansen, J.P. Gaze
Input for Mobile Devices by Dwell and Gestures. In
ETRA’12, 225-228, ACM Press, New York, 2012.
5. Heikkilä, H., and Räihä, K.-J. Simple Gaze Gestures and
the Closure of the Eyes as an Interaction Technique. In
ETRA’12, 147-154, ACM Press, New York, 2012.
6. Hoggan, E., Brewster, S. A., and Johnston, J.
Investigating the Effectiveness of Tactile Feedback for
Mobile Touchscreens. In CHI’08, 1573-1582, ACM
Press, 2008.
7. Hyrskykari, A., Istance, H. and Vickers, S. Gaze
Gestures or Dwell-Based Interaction. In ETRA’12, 229-
232, ACM Press, New York, 2012.
8. Isokoski, P. Text Input Methods for Eye Trackers Using
Off-Screen Targets. In ETRA’00, 15-21, ACM Press,
2000.
9. Istance, H., Hyrskykari, A., Immonen, L.,
Mansikkamaa, S., and Vickers, S. Designing Gaze
Gestures for Gaming: an Investigation of Performance.
In ETRA’10, 323-330, ACM Press, 2010.
10. Lukander, K., Jagadeesan, S., Chi, H., and Müller, K.
OMG! - A New Robust, Wearable and Affordable Open
Source Mobile Gaze Tracker. In MobileHCI’13, 408-
411, 2013.
11. Nielsen, J. Noncommand User Interfaces. In
Communications of the ACM, 36(4), 83-99, 1993.
12. Porta, M, and Turina, M. Eye-S: a Full-Screen Input
Modality for Pure Eye-based Communication. In
ETRA’08, 27-34, ACM Press, New York, 2008.
13. Rubine, D. Combining Gestures and Direct
Manipulation. In CHI’92, 659-660, ACM, 1992.
14. Stellmach, S., and Dachselt, R. Look & Touch: Gaze-
Supported Target Accuisition. In CHI’12, 2981-2990,
ACM, 2012.
15. Stellmach, S., and Dachselt, R. Still Looking:
Investigating Seamless Gaze-supported Selection,
Positioning, and Manipulation of Distant Targets. In
CHI’13, 285-294, ACM, 2013.
16. Wobbrock, J.O., Rubinstein, J., Sawyer, M.W., and
Duchowski, A.T. Longitudinal Evaluation of Discrete
Consecutive Gaze Gestures for Text Entry. In ETRA’08,
11-18, ACM Press, New York, 2008.
Figure 4. Boxplots of the answers (in scale 1 to 7) to the
question of “how comfortable” (left) and “how easy to use”
(right) was the given condition.
