ChapterPDF Available

“They see me scrollin”—Lessons Learned from Investigating Shoulder Surfing Behavior and Attack Mitigation Strategies

Authors:

Abstract

Mobile computing devices have become ubiquitous; however, they are prone to observation and reconstruction attacks. In particular, shoulder surfing, where an adversary observes another user’s interaction without prior consent, remains a significant unresolved problem. In the past, researchers have primarily focused their research on making authentication more robust against shoulder surfing—with less emphasis on understanding the attacker or their behavior. Nonetheless, understanding these attacks is crucial for protecting smartphone users’ privacy. This chapter aims to bring more attention to research that promotes a deeper understanding of shoulder surfing attacks. While shoulder surfing attacks are difficult to study under natural conditions, researchers have proposed different approaches to overcome this challenge. We compare and discuss these approaches and extract lessons learned. Furthermore, we discuss different mitigation strategies of shoulder surfing attacks and cover algorithmic detection of attacks and proposed threat models as well. Finally, we conclude with an outlook of potential next steps for shoulder surfing research.
“They see me scrollin”—Lessons Learned
from Investigating Shoulder Surfing
Behavior and Attack Mitigation
Strategies
Alia Saad, Jonathan Liebers, Stefan Schneegass, and Uwe Gruenefeld
1 Introduction
People interact with an evergrowing number of mobile computing devices in
everyday life. Nowadays, these devices have become ubiquitous and are com-
monly used in various places such as buses, trains, airports, coffee shops, and
restaurants [3, 14]. As a result of the continuous growth, privacy and security
challenges of these devices are becoming increasingly pressing. For example,
smartphones hold sensitive information about users, including business records,
financial interactions, personal details, and many more that should be kept hidden
from others. Nevertheless, finding privacy-preserving solutions is not restricted to
smartphones only. These solutions need to consider a variety of personal devices
(e.g., smartwatches and tablets) as well as public or shared devices (e.g., ATMs and
ticket machines).
All these devices are subject to various types of attacks. For instance, thermal
attacks, where intruders use thermal cameras to analyze the heat traces of the entered
authentication [1] or attacks that analyze the smudges on the screen for password
reconstruction and gaining illegitimate access [49,52]. However, smudge attacks are
mainly focused on the authentication period, and thermal attacks require technical
support and proper planning for a person to take a photo, feed it to a recognizer, and
gain unauthorized access. On the other hand, observation attacks, commonly known
as shoulder surfing attacks, are directly performed by humans and usually do not
require additional hardware to be successfully completed. Despite a large body of
work on these observation attacks, shoulder surfing remains a significant unresolved
problem that requires more attention.
A. Saad () · J. Liebers · S. Schneegass · U. Gruenefeld
University of Duisburg-Essen, Essen, Germany
e-mail: alia.saad@uni-due.de; jonathan.liebers@uni- due.de; stefan.schneegass@uni-due.de;
uwe.gruenefeld@uni-due.de
© The Author(s) 2023
N. Gerber et al. (eds.), Human Factors in Privacy Research,
https://doi.org/10.1007/978-3-031-28643- 8_10
199
200 A. Saad et al.
Fig. 1 Sketched example of a spontaneous shoulder surfing attack taking place during daily
commute
Observation attacks are not limited to a specific device, location, or acquaintance
level. Shoulder surfer can gaze at a person interacting with their personal phone or
at someone’s PIN, while they authenticate themselves after getting the phone out
of the pocket. They do not need an extra device and can quickly memorize entered
PINs or passwords. They could be standing in a train [46], or sitting next to the
victim in an office [2](seeFig.1). The incident could occur between two closely
tied people or with total strangers. Previous works confirm that observation attacks
are widespread and highly likely to occur [14].
With this pervasiveness, nearly everyone is both attacker and victim. Albeit,
recent studies showed that shoulder surfing incidents often take place opportunis-
tically, and without malicious intent. To this end, we consider a person looking at
the user’s interaction as an observer, as we are not sure of their motives. Many
researchers focused on understanding the occurrence of the observation attacks.
However, regardless of the intentions of the observers, researchers also worked on
various approaches to mitigate the risk of being observed, either by detection of the
observer, or by providing novel solutions to prevent the looker from perceiving the
content displayed.
Chapter Overview In the next section, we define the term shoulder surfing,
describe different dimensions relevant for shoulder surfing attacks, and present
key findings from previous research. Thereafter, we look at proposed strategies to
mitigate shoulder surfing attacks. Here, we start by looking at threat models and
algorithmic detection of shoulder surfers. Finally, we outline challenges and future
research directions for shoulder surfing research.
“They see me scrollin” 201
2 Investigating the Phenomenon
In this section, we first define shoulder surfing to set the scope for this chapter. After
that, we describe different methods with which researchers have investigated the
phenomenon and discuss their advantages and disadvantages. Finally, we highlight
the key findings from studies investigating shoulder surfing behavior.
2.1 Defining Shoulder Surfing (Attacks)
Observation attacks, commonly known as shoulder surfing attacks, are directly
performed by humans and usually do not require additional technology to be
successful. Farzand et al. [16] define shoulder surfing as observing someone’s
device screen without their consent. There are technology-based approaches to
investigate observation attacks using machine vision, commonly referred to as
recording attacks or video-based observation attacks (e.g., [30, 61]). Nonetheless,
this chapter primarily focuses on shoulder surfing attacks performed by humans.
To be classified as shoulder surfing, it does not matter if the motivation to
shoulder surf is simply curiosity or a deliberate attempt to steal information [9]. In
fact, shoulder surfing mainly occurs in an opportunistic, non-malicious way [14].
Nonetheless, failing to prevent bystanders from observing sensitive information
can lead to negative consequences such as financial loss, public exposure, and
embarrassment [3]. An example of a shoulder surfing attack is shown in Fig. 1.
In the following, we provide an overview of different dimensions that help
describe and classify shoulder surfing. The goal is not to present a complete
overview of all dimensions relevant to shoulder surfing but rather to discuss different
aspects that should be considered:
Motivation of Attack: Shoulder surfing attacks can be either intentional or
unintentional, whereas unintentional means in an opportunistic, non-malicious
way [9]. In most cases, shoulder surfing is unintentional and does not have
serious consequences [14]. Nonetheless, it can evoke negative feelings for both
parties and result in various coping strategies.
Attack Pattern: Shoulder surfing attacks can follow different attack patterns.
Abdrabou et al. [2] found three different patterns: continuous attacks, cautious
attacks, and repeated attack. While continuous attacks are characterized by
bystanders looking at the target device for an extended period with few or no gaze
shifts, cautious and repeated attacks alternate between observing the target device
and looking away. For the latter two, the difference is the victim’s behavior,
who either looks up from the target device (from time to time) or shows high
engagement. Friends, family, or colleagues at work may repeatedly observe their
peers and thereby combine multiple partial observations to form a hypothesis of
a target device’s secret [37, 57].
202 A. Saad et al.
Number of Attackers: In theory, a shoulder surfing attack can be performed by
multiple attackers. While some research considers threat models with more than
one attacker [24], many studies simplify this aspect and study 1:1 relationships
between victim and attacker.
Relationship Between Victim and Attacker: Besides the number of attackers, the
type of relationship (family, friend, colleague, stranger) is important as well.
Muslukhov et al. [37] conducted surveys and interviews to investigate users’
concerns about unauthorized access by insiders and strangers. They concluded
that observing unlock attempts, memorizing it, and thus gaining unauthorized
access by insiders are highly likely to occur. That is directly linked to insiders’
ability to observe interactions closely and repeatedly. Farzand et al. [16]showed
that the type of relationship impacts the choice of mitigation behavior. Moreover,
depending on the relationship with the attacker, victims often do not want them
to know they were caught.
Victim–Attacker Relative Pose: To successfully shoulder surf, the content on the
target device must be directly visible to the attacker (unless we reconstruct the
screen content from visual reflections with machine learning [60]). Thus, the
relative pose between victim and attacker is important, as the used term shoulder
surfing illustrates. A sitting pose, for example, enables shoulder surfing more
than a standing pose [46]. Furthermore, viewing angle and distance play an
important role as well [6]. However, tilting the device away from the observer,
a widely adopted defense strategy, provides limited protection from shoulder
surfing attacks [25].
Type of Device: Different devices can be the target of a shoulder surfing attack,
including but not limited to notebooks, tablets, smartphones, and smartwatches.
However, shoulder surfing can also occur when using shared devices or accessing
private information on public devices [9]. The main prerequisite for shoulder
surfing is that a bystander can observe the user’s screen. Hence, smartglasses are
unaffected and can be used as a mitigation strategy [58].
Type of Content: Mainly, two different types exist: (1) authentication-based and
(2) content-based shoulder surfing [18]. The primary focus of many shoulder
surfing studies is to investigate secure password or PIN entry [8]. While
authentication is, of course, important and prone to observational attacks, other
types of content can also be observed. Moreover, content-based shoulder surfing
is more frequently experienced than authentication-based shoulder surfing [18].
Previous work has examined different content types such as notifications, texts,
photos, social media, and gaming [6, 46]. Nevertheless, while different types of
content are affected by shoulder surfing, there are differences in their perceived
sensitivity [17].
Type of Environment: Shoulder surfing can take place in different environments
such as buses, trains, airports, coffee shops, and restaurants [3]. These environ-
ments can be classified in two different ways. One can either distinguish private,
semi-public (work), or public contexts [45] or differentiate between personal and
professional contexts [62]. Independent of the classification choice, the location
“They see me scrollin” 203
cannot be neglected when studying shoulder surfing attacks as it influences
victim and attacker behavior [48].
2.2 Research Methods
As outlined in the chapter “Empirical Research Methods in Usable Privacy and
Security” , privacy and security research has applied various methods. In this
section, we highlight the methods that were previously used to study shoulder
surfing. In summary, we classify these methods into four categories: (1) surveys and
interviews, (2) lab studies, (3) field/in-the-wild studies, and (4) studies in extended
reality. The following subsection describes the different methods and highlights
their advantages and disadvantages. Our goal is to provide an overview of the
different methods to support researchers and practitioners (new to the field) in
deciding which method to apply in their research.
Surveys and Interviews Surveys and interviews are helpful tools for privacy
researchers to gather valuable insights into a broader population or specific user
groups [36]. The difference between surveys and interviews is that in interviews,
a researcher takes an active role and directs questions to the interviewee (cf.,
Lazar et al.[27, 28]), while in surveys, a set of predefined questions is presented
to the participants. With surveys and interviews, it is possible to achieve various
objectives. On one side, researchers can use them to gather evidence for shoulder
surfing attacks in the real world and get insights into personal experiences with the
phenomenon from both victims and attackers of shoulder surfing incidents (e.g.,
[14]). On the other side, they help to understand preliminary performance metrics
of authentication techniques against observation attacks (e.g., robustness [4]) and
can even be used to quantify which parameters of these techniques help to make
them less observable (e.g., [54]). Different approaches to constructing surveys exist.
Noticeable is the inclusion of video material to present recreations of shoulder
surfing attacks to participants [4]. Aviv et al. [5] show that these videos embedded
in surveys can achieve results comparable to user studies in the lab.
Compared to other research methods, surveys allow larger sample sizes as
researchers can reach and recruit more participants. Nevertheless, sample sizes
vary enormously for shoulder surfing research. Previous work has reported studies
with more than 1000 participants (.n=1173)[4] to smaller numbers that remain
in the hundreds (e.g., .n=298 [54]or.n=174 [14]). Compared with other
research methods, surveys often report higher numbers of participants. Recently,
crowdsourcing platforms have entered the stage of privacy research and provide
researchers with access to different user groups (that can be specified concerning
various dimensions) [23]. Nowadays, researchers can more easily recruit a diverse
set of participants.
In addition to surveys, in-depth interviews can be a sensible next step that allows
scientists to understand the reasons behind the observed data [14]. Nonetheless,
204 A. Saad et al.
interviews can also be applied as a standalone method. For interviews, the more
active participation of a researcher asking questions can lead to more detailed
responses [28]. Moreover, interviews allow the live demonstration of specific
techniques under controlled conditions. For example, the interviewer can present
different shoulder surfing mitigation strategies to participants during the inter-
view [16].
Finally, there has been a recent study that explored shoulder surfing through a
longitudinal investigation, meaning they performed a diary study with 23 partici-
pants over one month [18]. They found that content-based shoulder surfing takes
place more frequently than authentication-based shoulder surfing.
While we presented different methods in this part, they all have in common
that they rely on self-reporting. While self-reporting is frequently deployed in
privacy research, it has a few noteworthy drawbacks. As researchers do not directly
observe a phenomenon, factor, or effect, they rely on the subjective perception of
the participant, which can include a recall bias [43]. Moreover, not every type
of information can be gathered with self-reporting; however, asking indirect and
anonymity-preserving questions can minimize social desirability bias [33, 53].
Lab Studies Scientists often conduct experiments to answer their research
questions concerning shoulder surfing. In experiments, it is often necessary that
researchers can observe a shoulder surfing situation taking place. Due to the
challenges of researching the phenomenon during field or in-the-wild studies (see
below), these studies are primarily carried out in the lab. Moreover, compared to
surveys and interviews, recruiting participants is more difficult, and conducting
the experiment is often more workload-intense. As a result, experiments generally
report smaller sample sizes. Nevertheless, a lab study also has certain advantages,
for example, compared to field or in-the-wild studies. The most significant benefit
(compared to other study types) is the high degree of control over the experimental
conditions. Moreover, a lab study allows gathering consent from all involved parties
before the experiment.
When conducting a lab study to research different dimensions of a shoulder
surfing attack (e.g., the resilience of authentication techniques against human
shoulder surfers), a challenge is to replicate these attacks for the study [56]. In lab
studies, participants often take over the role of the attacker (e.g., [46]). Nevertheless,
it remains challenging to replicate realistic attacks, as often they are performed
out of boredom in opportunistic moments [14]. Simply instructing participants to
perform a shoulder surfing attack would broadly differ from the behavior observable
during an actual attack. To overcome this challenge, researchers have designed
studies that inform participants about the study’s goals toward the end (e.g., [46]).
These studies partially deceive participants by leaving out specific study details not
to influence their behavior. However, it should be noted that deceiving participants in
a user study can be problematic and not justified. Hence, it is strongly encouraged to
balance ethical implications and knowledge gain and act cautiously when deceiving
participants.
“They see me scrollin” 205
A different approach is to research factors and effects that are not related to the
timing, occurrence, or behavior of shoulder surfing attacks but instead focus on
aspects that can be researched with the research goal out in the open. For example,
a previous study has investigated the effect viewing angle and distance have on
the success of shoulder surfing attacks [6]. Here, a lab study can offer control to
isolate research factors from others that would introduce too much complexity to
the experiment.
In-the-Wild or Field Studies Researching the phenomenon of shoulder surfing
with in-the-wild or field studies sheds more light on the contexts in which these
attacks take place and could provide insights into the behavior of attackers and
victims. However, performing these studies is very challenging and, thus, rarely
conducted. One of these studies was a two-week in-the-wild study conducted by
Schneegass et al. [48], where they investigated the likelihood of shoulder surfing
attacks occurrence during unlock events. Nonetheless, shoulder surfing is socially
unacceptable and privacy-invasive. Hence, observing these attacks requires consent,
potentially biasing participants and making it very difficult to observe authentic
interactions. Moreover, outside the lab, bystanders get involved quickly; when
that happens, their consent is also necessary (e.g., when recording video for eye
tracking). In the past, researchers have primarily relied on surveys and interviews to
assess in-the-wild experiences [14], relying on self-assessment as the most frequent
research method. To encompass both the benefits of a study in the lab (such as its
associated high degree of control) and to enable researching more realistic (in situ)
shoulder surfing scenarios, researchers have applied eXtended Reality as a study
method.
Studies in Extended Realities Recently, eXtended Reality (XR) [42] entered
human–computer interaction (HCI) as a means to conduct user studies that are not
directly related to XR but use XR as a modality to conduct user studies instead (e.g.,
[31]). This is particularly the case for user studies that are taking place in virtual
reality (VR) in a virtual environment (VE), whereas XR could implicate “augmented
reality” (AR) or “mixed reality” (MR) as well. The trend of using XR as a research
method got amplified with the ongoing Covid-19 pandemic as different frameworks
appeared [19, 40].
Using VR to research the shoulder surfing phenomenon has several inherent
benefits. First, a virtual environment allows a more believable recreation of a real-
life situation, which would otherwise be hard to recreate in the lab (e.g., a bus stop
or office environment with different people present [2]; see Fig. 2). In addition to
the realistic recreated scenes, VR allows maintaining the consistency among study
participants, avoiding external uncontrolled situations. With eye trackers embedded
in the head-mounted displays (HMD), researchers are able to capture and analyze
the gaze of the participants. Accordingly, they are able to profoundly understand
the observation attacks cycles and expect what triggers the observers’ attention. As
VR is associated with a high degree of immersion, it allows placing the subject in a
simulated, virtual environment, where they can experience the situation as intended
206 A. Saad et al.
Fig. 2 Example taken from a previous paper that studied shoulder surfing in virtual reality [2].
The figure shows two virtual scenes that were used to investigate observing others’ displays in an
open office space (left) and a bus stop (right). The read markers indicate the participants’ initial
position
by the researchers. Here, the degree of presence can be assessed through the usage
of presence questionnaires [50, 51, 59].
Potentially, such studies can also run outside the lab on HMDs owned by
participants [40], and they were validated for usable security evaluations [35].
Additionally, user studies in XR allow fulfilling particular requirements specific for
shoulder surfing studies. One is privacy, as conducting a user study in a real-world
environment with real victims can be considered ethically challenging, whereas
shoulder surfing a virtual avatar in a virtual environment (VE) is less likely an
issue. Furthermore, conducting a user study in a VE allows for a very high degree
of control since the environment is simulated by a computer, often exceeding the
capability of control that an experimenter has over a real-world situation, even if
it takes place in a lab. The high degree of control allows for replicability of such
user studies between participants, as the experienced situation can be made to be
precisely always the same.
2.3 Key Findings on Shoulder Surfing Behavior
With the growing number of studies investigating shoulder surfing events, we
highlight the key findings on observers behaviors that we believe are of high
relevance.
Observations Are Often More Random Than Planned In the survey by Eiband
et al. [14], the main findings showed that despite the fact that observations are
frequently conducted on an opportunistic basis, they go beyond exposing the
authentication. Several participants reported negative feelings when other content
such as personal photos or texts are exposed.
Victim–Attacker Pose Relationships Are Unalike In 2021, Saad et al. [46]
explored the tendency of bystanders to shoulder surf in a scenario within an
underground train. To that end, they varied the point of view of the attacker (standing
vs. sitting) and the position of the victim (again standing vs. sitting) and used a
“They see me scrollin” 207
Fig. 3 User study conducted in virtual reality to investigate shoulder surfing attacks with
prerecorded 360.
videos [46]. Left to right: viewpoints of the participants with four different
relative poses to the (virtual) victim: standing to standing, standing to sitting, sitting to standing,
and sitting to sitting
360.
camera to obtain a photorealistic recording of this setting, where several actors
played either the role of the victim or became extras to simulate other people on
the train. This recording then was played back to participants in a user study on an
HMD that was equipped with an eye tracker in a lab study, and the point of view
of the participants is seen in Fig. 3. Through the eye-tracking data, it was apparent
that participants gazed at the object of interest, a smartphone held by the victim, and
11.16% of the time they were nearby.
VR Reflects Genuine Behavior. . . In 2022, Abdrabou et al. [2] conducted another
project on the understanding of shoulder surfer behavior and the associated attack
patterns. Here, they created a simulation in virtual reality with virtual, human-
like avatars who were either located at a bus stop scene or within an office. The
human participant of this study then was placed inside this VE through a VR HMD,
which was again equipped with an eye tracker. The experimenters then recorded
the participants’ gaze and their walking patterns in VR and found that participants
looked at several objects of interest (e.g., smartphones in the bus stop scene or
monitors in the office scene) 5.7 times on average, whereas the average eye contact
duration was 1.61 s.
...but Immersion Is Needed. Also in 2022, Mathis et al. [34] considered the
differences between non-immersive and immersive VR for shoulder surfing research
and conducted a user study to explore the characteristics of both settings. They
considered shoulder surfing attacks on automated teller machines, smartphone
personal identification numbers (PIN), and smartphone pattern unlock mechanisms.
They compare three scenarios, 2D video observations, 3D observations, and VR
observations. The first scenario, 2D video observations, consists of the study
participants watching a video of the shoulder surfing situation that they cannot
influence on a traditional computer monitor, whereas in 3D observations, they
could use the keyboard and mouse to walk around. These two conditions then
were compared against each other and VR observations, where participants were
wearing a VR headset and could freely move around and adjust their observation
perspective. The authors found that VR observations lead to a significantly higher
sense of presence and involvement and that VR observations also lead to the most
accurate shoulder surfing observations.
208 A. Saad et al.
There Is More than Smartphones There are other devices that are becoming
more ubiquitous nowadays, smartwatches for instance. Recently, more studies
are proposing authentication approaches for smartwatches, with resilience against
shoulder surfing as a key metric for robustness [38, 39].
In conclusion, we can observe that there is an increasing number of publications
that utilize XR, particularly VR, as a research method for shoulder surfing research.
The high degree of immersion lets the participants of a user study easily take the
role of the attacker, while such a lab study setting allows for an efficient resolution
of the problematic aspects connected to ethics in this kind of research. Furthermore,
VR allows the study to be exactly the same for each subject, as the computer-driven
simulation creates an easily repeatable environment. Thereby, realistic scenarios can
effectively be replicated in the lab.
3 Mitigating Shoulder Surfing Attacks
For the mitigation of shoulder surfing attacks, it is important to note that not
every shoulder surfing incident is equally problematic. One important aspect to
consider is the type of content visible. For content-based shoulder surfing, we
need to understand what is considered sensitive content as it plays an important
role in selecting a suitable mitigation strategy. To tackle this challenge, Farzand et
al. [17] present a typology of perceived sensitivity that can help to understand the
content sensitivity. Furthermore, one needs to take into account that the perception
of shoulder surfing is different between cultures [47]. As a consequence, it also
differs what is considered sensitive content.
In the following section, we look at research that aims to find solutions to mitigate
shoulder surfing attacks. Therefore, we start by looking at different threat models
against which researchers and practitioners can evaluate their mitigation strategies.
After that, we briefly describe technical approaches to detect shoulder surfing and
their current limitations. Finally, we present an overview of different mitigation
strategies.
3.1 Threat Models
Threat models provide a systematic approach to investigate potential weaknesses to
privacy and security [32]. For shoulder surfing, different threat models have been
considered in the literature. Below, we provide a selection of these models and
describe them briefly. It should be noted that also mixes of these are possible (e.g.,
a repeated attack that is technology-supported [7]):
“They see me scrollin” 209
Weak Attacks: A shoulder surfing attack is considered a weak attack if it is
performed by a human observer without the help of any technology and with
only limited practice [11].
Trained Shoulder Surfers. Compared to weak attacks, trained shoulder surfers are
more effective by training themselves. They often employ cognitive strategies
that help to reach higher success rates [26]. Please note that trained shoulder
surfers manage to be more effective without using recording devices.
Repeated Attacks: The repeated attacks threat model assumes that an attacker can
repeatedly observe the target device of the victim. Moreover, this threat model
often considers the attacker to be at close range—the attacker quite literally looks
over the victims’ shoulder [7].
Insider Attacks: Quite similar to the repeated attacks threat model are the insider
attacks. The main difference is that for this type of attack, family, friends,
or colleagues perform them. They may repeatedly observe the victim, and by
combining these partial observations, it is easier to form a hypothesis on the
victim’s secret [57].
Multiple Attackers: The shoulder surfing attacks become more threatening when
multiple attackers try to observe the target device. In this case, attackers can
coordinate by either focusing on specific parts or organizing distraction and
information stealing roles between attackers [24].
Technology-Supported Attacks: The probably strongest form of shoulder surfing
attacks are technology-supported ones. In these cases, an attacker is recording
the victim’s interactions, for example, when drawing money from an ATM [10].
With recent technology advances, camera-based sensors can be manufactured
in very tiny proportions, allowing attackers to seamlessly integrate them in
their clothing or accessories. When analyzing the recorded data with machine
learning, breaches of privacy are possible even when the attacker is not direct
line of sight because reflections on glasses are sufficient for reconstruction of
screen content [60].
3.2 Algorithmic Detection of Attacks
To mitigate shoulder surfing attacks, they first need to be detected. In previous
research, detecting shoulder surfing attacks is primarily achieved by focusing on the
human attackers. Here, algorithmic approaches oftentimes rely on visual sensor data
(i.e., monochrome and RGB cameras). As shoulder surfing is frequently researched
for mobile devices, the built-in camera is a good source for visual information
to detect attackers. For example, Ali et al. [3] investigated the use of the built-
in camera on mobile devices to detect if an unauthorized person tries to gain
access to the device. Here, to detect an observer, face detection is applied to the
incoming video feed. Interestingly, popular operating systems such as Android
come with real-time face detection capabilities that can be used for detecting
210 A. Saad et al.
Fig. 4 Study apparatus to investigate the influence of distance and viewing angle on shoulder
surfing success rate, figure taken from Bâce et al. [6] licensed under CC BY-NC-ND 4.0. The
subfigures show examples of different content types on the phone display: (left to right): text, PIN,
photo, and no content visible. The mechanical prototype visible rotated the smartphone between 0,
30, and 60.
shoulder surfers [7]. Nonetheless, not every detected face is necessarily a potential
attacker as other factors play an important role as well, such as gaze direction and
context, among others. In a recent study, different angles and distances have been
investigated to understand which of them are most critical as they provide a good
position for shoulder surfing [6]. The threat model was also based on evaluating
people’s perception on the displayed content that varied between visual, textual,
and authentication, as seen in Fig. 4.
Nevertheless, visual detection of potential shoulder surfing also comes with a
few downsides. First, they require the camera to be active and to record the scene.
This scene likely involves the users of the device as well and, thereby, introduces
another privacy risk. Furthermore, not only the privacy of a user may be violated, but
also that of bystanders (as it continuously records the scene). Another issue is that
the continuous recording and processing of the video feed drains the battery more
quickly [7]. Hence, researchers have explored other options as well. For example,
Lian et al. [29] used “multiple sensors, i.e., video camera module, ultrasonic
distance module, light sensor module, to detect screen peeping, user distance and
environmental lightness.” Here, future studies should compare the different sensor
technologies and develop adaptive strategies that take the context into consideration.
For example, when a user is logged in to their wireless network at home and no other
Bluetooth signatures are around, continuous monitoring via the built-in camera to
detect shoulder surfing may not be necessary.
3.3 Prevention Strategies
Oftentimes, a detection algorithm proposed by researchers goes hand in hand
with an implementation of a mitigation strategy (cf. [44]). In the following, we
discuss two different strategy types into which proposed systems can be classified.
“They see me scrollin” 211
On one side, there are strategies that try to be generalizable toward every kind
of content, and on the other side, there are strategies that focus on mitigating
attacks against specific types of contents. These two strategies are in line with
how we categorize shoulder surfing attacks into authentication-based and content-
based shoulder surfing. Here, it is important to note that while authentication-based
shoulder surfing is perceived as more problematic, content-based shoulder surfing
is occurring more frequently [18].
Strategies Independent of Content Often times, researchers propose systems that
mitigate shoulder surfing attacks independent of the content shown by the target
device. Different systems have been proposed that try to create awareness for an
actively ongoing shoulder surfing attack. For example, Ali et al. [3] proposed a
system that informs users whether text on the screen could be read by an attacker. To
better understand, in which way users want to be alerted, researchers have conducted
a user study to compare four different methods: vibro-tactile, front LED, on-screen
icons, and video feedback, finding that vibro-tactile feedback works best, as seen in
Fig. 5. Their findings showed that vibration feedback allowed for a faster response
time, in comparison to the other three methods [44]. Moreover, it has been examined
how additional parameters such as distance and orientation can benefit victims in
applying appropriate actions [62].
While awareness-based systems leave it to the user to decide on how they
want to react, researchers have proposed different strategies that help users in their
actions [9] or automatically react to shoulder surfing attacks [29]. Here, users can
either move or hide information presented on the screen by performing explicit
interactions [9] or information is automatically masked [9, 29] (e.g., with the help
of eye tracking [41]). Lian et al. [29] found that with limited brightness or contrast,
only the user could read the screen, while others have trouble reading it [29].
Furthermore, different strategies have been proposed that do not rely upon
detecting a shoulder surfer at first, but rather are applied constantly. For example,
Chen et al. [12] developed Hide Screen, which utilizes human vision characteristics
to preserve privacy. Simplified, the approach allows changing the readability of
information based on the viewing angle. Instead of hiding the information from an
attacker, Watanabe et al. [55] suggest adding additional information that is designed
to throw an attacker off. They suggest showing multiple cursors on the screen
Fig. 5 Different feedback conditions to communicate a shoulder surfing incident investigated in
previous work [44]. The different feedback conditions are (from left to right): (1) front LED,
(2) video preview, (3) vibro-tactile, and (4) on-screen icon. The authors found that vibro-tactile
feedback results in the lowest reaction time
212 A. Saad et al.
and, thereby, effectively hiding the real cursor for an observer. Finally, it has been
proposed to extend an observable screen with a second screen that is not observable
and can be used to show private information. For example, Winkler et al. [58]are
using smartglasses to show private information that would have otherwise be shown
on the smartphone display.
Strategies Focused on Specific Types of Content Because not every type of
content requires the same level of protection, many proposed strategies that are
highly dependent on the type of content that they protect. In particular, authen-
tication approaches need high protection against shoulder surfing attacks. Hence,
researchers have suggested a variety of authentication techniques that are more
resilient against observational attacks.
Bianchi et al. [7] proposed to use a composition of non-visual cues (i.e.,
audio and haptic cues) to enter a password. As a result, an observational attack
cannot rely on visual information only to decipher the password. Furthermore,
others have suggested to use gaze as an input modality in combination with
graphical passwords [10]. Thereby, an attacker would need to observe the eye
gaze of the victim additionally to the phone screen, making it very challenging
to reconstruct the password. Another strategy is to extend the input surface for
the authentication scheme toward the backside of the smartphone, which is more
difficult to observe [13].
Besides authentication approaches, researchers have focused on other types of
content. For example, Eiband et al. [15] have investigated how text can be presented
in a way that is readable to the user but unreadable to an observer. In essence, they
propose to display text in the user’s own handwriting. While this does not prevent
an attacker from reading the text, it significantly slows them down.
4 Challenges and Future Research Directions
In the following, we present challenges and research directions concerning the
methodology of researching shoulder surfing and the phenomenon itself. These
are particularly related to the methodology of shoulder surfing research and the
attacker’s behavior.
Research Methods to Investigate Shoulder Surfing While conducting research
on shoulder surfing in the wild, several challenges regarding the methodology
became apparent. First of all, a central element is an ethical dilemma associated
with the necessity of obtaining the shoulder surfer’s consent. When researchers
ethically design an experiment on shoulder surfing that involves participants,
participants usually have to get into the role of either the victim or attacker. However,
shoulder surfing usually is an interaction that is very affective by its nature [14],
hence instructing participants on the roles that they should get into highly inflects
their behavior, and thus, results elicited from the study. Consequently, there is a
dichotomy between asking for consent and subjects’ unchanged behavior that needs
“They see me scrollin” 213
to be weighed individually for each study, taking the objectives of the study into
account.
Another argument on shoulder surfing studies is to simultaneously consider both
roles of the attacker and the victim. Considering only the role of the observer and
not the victim could leave out vital parts of the shoulder surfing incident, such as
the occlusion of the phone display by the victim [6].
Virtual Reality for User Studies To overcome some of the challenges related to
this ethical dichotomy, several research projects utilized virtual reality to simulate
the shoulder surfing interaction with virtual avatars [2, 34, 46]. Although it is not
necessary to obtain consent from a virtual avatar that has the role of the victim,
it, however, still is necessary to obtain consent from a participant that gets into
the role of the attacker. Furthermore, virtual reality allows for a simulation of the
environment; hence, the interaction can be explored in different settings that would
be hard to replicate in a physical lab.
However, virtual reality is also only a limited solution, as there are certain
aspects impacted by the simulation of the environment. For example, today’s head-
mounted displays can influence people’s behavior such as their movement [20]or
also their social comfort distance that is less in virtual reality than in reality [22].
They can, however, help in recreating scenarios from the real-world by simulating
them in a lab, as conducting field studies or in-the-wild experiments is particularly
challenging due to the ethical aspects, particularly, when uninvolved third parties
become part of the investigation. The same applies to other methodologies such
as the usage of recording videos outside the lab, the so-called “lifelogs”, as using
cameras impacts the protection of private information of both the wearer and
potential bystanders [21].
Identifying Sensitive Content In general, two types of shoulder surfing are
distinguished: authentication-based and content-based shoulder surfing. While
authentication-based shoulder surfing is inherently problematic as it exposes
sensitive information (e.g., PIN or password), it is more complicated for content-
based shoulder surfing that happens more frequently [18]. Privacy is an individual
concept. Hence, what one person considers sensitive information may not be
considered sensitive by someone else. This makes it very difficult to have an overall
solution that equally protects all users. As a consequence, we need to investigate
what content is considered sensitive (e.g., [17]). Furthermore, we need to examine
different factors that can influence the perception of what is considered sensitive
content such as cultural differences [47].
Understanding the Attacks and Behavior Another open research direction is to
create an understanding of the shoulder surfing interaction itself, by, for instance,
creating models of it. Here, Abdrabou et al. have created one of the first works in
creating a model of attack patterns [2]. Their study took place in virtual reality;
hence, creating a model-based understanding of the phenomenon, in reality, is still
an open research opportunity nowadays. It is therefore necessary to conduct further
studies to determine more substance to derive models about behavior within more
214 A. Saad et al.
contexts of the interaction. This includes, but is not limited to, in-the-wild studies
as well as long-term studies to understand, whether the behavior changes over time.
Additionally, recent studies focus on password attacks but do not have a strong
focus on understanding shoulder surfing behavior in general [8]. However, when
considering only the attacks on passwords, such as android pattern locks, models
were already created that predict the grade of observability [54]. This also opens up
the opportunity to further explore the type of content that is particularly attracting
shoulder surfing attacks, which partly has been covered by recent studies [2, 46].
5 Conclusion
In this chapter, we presented lessons learned from research on the shoulder surfing
phenomenon and attack mitigation strategies. We started with a definition of
shoulder surfing and an introduction of different types of attacks. After that, we
present different research methods that have been applied in the past and discussed
key findings related to shoulder surfing behavior. Next, we gave an overview of
different threat models for shoulder surfing and discussed algorithmic detection of
these attacks and different mitigation strategies. We concluded the chapter with an
outlook on persistent challenges and future research directions. We believe that this
book chapter offers a great starting point for new researchers and practitioners in
the field. Moreover, we see great potential for eXtended Reality to overcome the
limitations that field and in-the-wild studies introduce.
References
1. Abdelrahman, Y., Khamis, M., Schneegass, S., & Alt, F. (2017). Stay cool! Understanding ther-
mal attacks on mobile-based user authentication. In Proceedings of the 2017 CHI Conference
on Human Factors in Computing Systems (pp. 3751–3763).
2. Abdrabou, Y., Rivu, S. R., Ammar, T., Liebers, J., Saad, A., Liebers, C., Gruenefeld, U.,
Knierim, P., Khamis, M., Makela, V., Schneegass, S., & Alt, F. (2022). Understanding shoulder
surfer behavior and attack patterns using virtual reality. In P. Bottoni & E. Panizzi, (Eds.),
Proceedings of the 2022 International Conference on Advanced Visual Interfaces (pp. 1–9).
ACM.
3. Ali, M. E., Anwar, A., Ahmed, I., Hashem, T., Kulik, L., & Tanin, E. (2014). Protecting
mobile users from visual privacy attacks. In Proceedings of the 2014 ACM International
Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication,UbiComp
’14 Adjunct (pp. 1–4). Association for Computing Machinery.
4. Aviv, A. J., Davin, J. T., Wolf, F., & Kuber, R. (2017). Towards baselines for shoulder surfing
on mobile authentication. In Proceedings of the 33rd Annual Computer Security Applications
Conference (pp. 486–498).
5. Aviv, A. J., Wolf, F., & Kuber, R. (2018). Comparing video based shoulder surfing with live
simulation. In Proceedings of the 34th Annual Computer Security Applications Conference,
ACSAC ’18 (pp. 453–466). Association for Computing Machinery.
“They see me scrollin” 215
6. Bâce, M., Saad, A., Khamis, M., Schneegass, S., & Bulling, A. (2022). PrivacyScout: Assess-
ing vulnerability to shoulder surfing on mobile devices. Proceedings on Privacy Enhancing
Technologies, 1, 21.
7. Bianchi, A., Oakley, I., Kostakos, V., & Kwon, D. S. (2010). The phone lock: Audio and haptic
shoulder-surfing resistant PIN entry methods for mobile devices. In Proceedings of the Fifth
International Conference on Tangible, Embedded, and Embodied Interaction, TEI ’11 (pp.
197–200). Association for Computing Machinery.
8. Bošnjak, L., & Brumen, B. (2020). Shoulder surfing experiments: A systematic literature
review. Computers & Security, 99, 102023.
9. Brudy, F., Ledo, D., Greenberg, S., & Butz, A. (2014). Is anyone looking? Mitigating
shoulder surfing on public displays through awareness and protection. In Proceedings of
The International Symposium on Pervasive Displays, PerDis ’14 (pp. 1–6). Association for
Computing Machinery.
10. Bulling, A., Alt, F., & Schmidt, A. (2012). Increasing the security of gaze-based cued-recall
graphical passwords using saliency masks. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems, CHI ’12 (pp. 3011–3020). Association for Computing
Machinery.
11. Chakraborty, N., & Mondal, S. (2014). An improved methodology towards providing immunity
against weak shoulder surfing attack. In A. Prakash & R. Shyamasundar (Eds.), Information
Systems Security (pp. 298–317). Springer International Publishing.
12. (Daniel) Chen, C.-Y., Lin, B.-Y., Wang, J., & Shin, K. G. (2019). Keep others from peeking at
your mobile device screen! In The 25th Annual International Conference on Mobile Computing
and Networking, MobiCom ’19. Association for Computing Machinery.
13. De Luca, A., Harbach, M., von Zezschwitz, E., Maurer, M.-E., Slawik, B. E., Hussmann, H., &
Smith, M. (2014). Now you see me, now you don’t: Protecting smartphone authentication from
shoulder surfers. In Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems, CHI ’14 (pp. 2937–2946). Association for Computing Machinery.
14. Eiband, M., Khamis, M., Von Zezschwitz, E., Hussmann, H., & Alt, F. (2017). Understanding
shoulder surfing in the wild: Stories from users and observers. In Proceedings of the 2017 CHI
Conference on Human Factors in Computing Systems (pp. 4254–4265).
15. Eiband, M., von Zezschwitz, E., Buschek, D., & Hußmann, H. (2016). My scrawl hides it all:
Protecting text messages against shoulder surfing with handwritten fonts. In Proceedings of the
2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems,CHIEA
’16 (pp. 2041–2048). Association for Computing Machinery.
16. Farzand, H., Bhardwaj, K., Marky, K., & Khamis, M. (2021). The interplay between personal
relationships & shoulder surfing mitigation. In Mensch Und Computer 2021, MuC ’21 (pp.
338–343). Association for Computing Machinery.
17. Farzand, H., Marky, K., & Khamis, M. (2022). “I hate when people do this; there’s a lot
of sensitive content for me”: A typology of perceived privacy-sensitive content in shoulder
surfing scenarios. In Proceedings of the Eighteenth USENIX Conference on Usable Privacy
and Security. USENIX Association.
18. Farzand, H., Marky, K., & Khamis, M. (2022). Shoulder surfing through the social lens:
A longitudinal investigation & insights from an exploratory diary study. In 2022 European
Symposium on Usable Security (pp. 85–97).
19. Gruenefeld, U., Auda, J., Mathis, F., Schneegass, S., Khamis, M., Gugenheimer, J., & Mayer, S.
(2022). VRception: Rapid prototyping of cross-reality systems in virtual reality. In Proceedings
of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22. Association
for Computing Machinery.
20. Hollman, J. H., Brey, R. H., Robb, R. A., Bang, T. J., & Kaufman, K. R. (2006). Spatiotemporal
gait deviations in a virtual reality environment. Gait & Posture, 23(4), 441–444.
21. Hoyle, R., Templeman, R., Anthony, D., Crandall, D., & Kapadia, A. (2015). Sensitive lifelogs:
A privacy analysis of photos from wearable cameras. In Proceedings of the 33rd Annual ACM
Conference on Human Factors in Computing Systems, CHI ’15 (pp. 1645–1648). Association
for Computing Machinery.
216 A. Saad et al.
22. Iachini, T., Coello, Y., Frassinetti, F., Senese, V. P., Galante, F., & Ruggiero, G. (2016).
Peripersonal and interpersonal space in virtual and real environments: Effects of gender and
age. Journal of Environmental Psychology, 45, 154–164.
23. Jin, H., Shen, H., Jain, M., Kumar, S., & Hong, J. I. (2021). Lean privacy review: Collecting
users’ privacy concerns of data practices at a low cost. ACM Transactions on Computer-Human
Interaction, 28(5), 1–55.
24. Khamis, M., Bandelow, L., Schick, S., Casadevall, D., Bulling, A., & Alt, F. (2017). They are
all after you: Investigating the viability of a threat model that involves multiple shoulder surfers.
In Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia,
MUM ’17 (pp. 31–35). Association for Computing Machinery.
25. Khan, H., Hengartner, U., & Vogel, D. (2018). Evaluating attack and defense strategies for
smartphone PIN shoulder surfing. In Proceedings of the 2018 CHI Conference on Human
Factors in Computing Systems, CHI ’18 (pp. 1–10). Association for Computing Machinery.
26. Kwon, T., Shin, S., & Na, S. (2014). Covert attentional shoulder surfing: Human adversaries are
more powerful than expected. IEEE Transactions on Systems, Man, and Cybernetics: Systems,
44(6), 716–727.
27. Lazar, J., Feng, J. H., & Hochheiser, H. (2017). Chapter 5: Surveys. In J. Lazar, J. H. Feng, & H.
Hochheiser (Eds.), Research methods in human computer interaction (2nd ed., pp. 105–133).
Morgan Kaufmann.
28. Lazar, J., Feng, J. H., & Hochheiser, H. (2017). Chapter 8: Interviews and focus groups. In J.
Lazar, J. H. Feng, & H. Hochheiser (Eds.), Research methods in human computer interaction
(2nd ed., pp. 187–228). Morgan Kaufmann.
29. Lian, S., Hu, W., Song, X., & Liu, Z. (2013). Smart privacy-preserving screen based on multiple
sensor fusion. IEEE Transactions on Consumer Electronics, 59(1), 136–143.
30. Maggi, F., Volpatto, A., Gasparini, S., Boracchi, G., & Zanero, S. (2011). Poster: Fast,
automatic iPhone shoulder surfing. In Proceedings of the 18th ACM Conference on Computer
and Communications Security (pp. 805–808).
31. Mäkelä, V., Radiah, R., Alsherif, S., Khamis, M., Xiao, C., Borchert, L., Schmidt, A., & Alt,
F. (2020). Virtual field studies: Conducting studies on public displays in virtual reality. In R.
Bernhaupt, F. F. Mueller, D. Verweij, J. Andres, J. McGrenere, A. Cockburn, I. Avellino, A.
Goguey, P. Bjørn, S. Zhao, B. P. Samson, & R. Kocielnik (Eds.), Proceedings of the 2020 CHI
Conference on Human Factors in Computing Systems (pp. 1–15). ACM.
32. Marback, A., Do, H., He, K., Kondamarri, S., & Xu, D. (2013). A threat model-based approach
to security testing. Software: Practice and Experience, 43(2), 241–258.
33. Marques, D., Guerreiro, T., & Carriço, L. (2014). Measuring snooping behavior with surveys:
It’s how you ask it. In CHI ’14 Extended Abstracts on Human Factors in Computing Systems,
CHI EA ’14 (pp. 2479–2484). Association for Computing Machinery.
34. Mathis, F., O’Hagan, J., Khamis, M., & Vaniea, K. (2022). Virtual reality observations: Using
virtual reality to augment lab-based shoulder surfing research. In 2022 IEEE Conference on
Virtual Reality and 3D User Interfaces (VR) (pp. 291–300). IEEE.
35. Mathis, F., Vaniea, K., & Khamis, M. (2021). RepliCueAuth: Validating the use of a lab-
based virtual reality setup for evaluating authentication systems. In Y. Kitamura, A. Quigley,
K. Isbister, T. Igarashi, P. Bjørn, & S. Drucker (Eds.), Proceedings of the 2021 CHI Conference
on Human Factors in Computing Systems (pp. 1–18). ACM.
36. Müller, H., Sedley, A., & Ferrall-Nunge, E. (2014). Survey research in HCI. In Ways of knowing
in HCI (pp. 229–266). Springer.
37. Muslukhov, I., Boshmaf, Y., Kuo, C., Lester, J., & Beznosov, K. (2013). Know your enemy: The
risk of unauthorized access in smartphones by insiders. In Proceedings of the 15th International
Conference on Human-Computer Interaction with Mobile Devices and Services (pp. 271–280).
38. Nagatomo, M., Watanabe, K., Aburada, K., Okazaki, N., & Park, M. (2019). Proposal and
evaluation of authentication method having shoulder-surfing resistance for smartwatches using
shift rule. In International Conference on Network-Based Information Systems (pp. 560–569).
Springer.
“They see me scrollin” 217
39. Park, M., Aburada, K., & Okazaki, N. (2021). Proposal and evaluation of a gesture authenti-
cation method with peep resistance for smartwatches. In 2021 Ninth International Symposium
on Computing and Networking Workshops (CANDARW) (pp. 359–364). IEEE.
40. Radiah, R., Mäkelä, V., Prange, S., Rodriguez, S. D., Piening, R., Zhou, Y., Köhle, K., Pfeuffer,
K., Abdelrahman, Y., Hoppe, M., Schmidt, A., & Alt, F. (2021). Remote VR studies: A
framework for running virtual reality studies remotely via participant-owned HMDs. ACM
Transactions on Computer-Human Interaction, 28(6), 1–36.
41. Ragozin, K., Pai, Y. S., Augereau, O., Kise, K., Kerdels, J., & Kunze, K. (2019). Private reader:
Using eye tracking to improve reading privacy in public spaces. In Proceedings of the 21st
International Conference on Human-Computer Interaction with Mobile Devices and Services,
MobileHCI ’19. Association for Computing Machinery.
42. Rauschnabel, P. A., Felix, R., Hinsch, C., Shahab, H., & Alt, F. (2022). What is XR? Towards
a framework for augmented and virtual reality. Computers in Human Behavior, 133, 107289.
43. Robins, R. W., Fraley, R. C., & Krueger, R. F. (2009). Handbook of research methods in
personality psychology. Guilford Press.
44. Saad, A., Chukwu, M., & Schneegass, S. (2018). Communicating shoulder surfing attacks
to users. In Proceedings of the 17th International Conference on Mobile and Ubiquitous
Multimedia, MUM 2018 (pp. 147–152). Association for Computing Machinery.
45. Saad, A., Gruenefeld, U., Mecke, L., Koelle, M., Alt, F., & Schneegass, S. (2022). Mask
removal isn’t always convenient in public!—The impact of the Covid-19 pandemic on device
usage and user authentication. In Extended Abstracts of the 2022 CHI Conference on Human
Factors in Computing Systems, CHI EA ’22. Association for Computing Machinery.
46. Saad, A., Liebers, J., Gruenefeld, U., Alt, F., & Schneegass, S. (2021). Understanding
bystanders’ tendency to shoulder surf smartphones using 360-degree videos in virtual reality.
In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction
(pp. 1–8). ACM.
47. Saleh, M., Khamis, M., & Sturm, C. (2019). What about my privacy, Habibi?. In D. Lamas, F.
Loizides, L. Nacke, H. Petrie, M. Winckler, & P. Zaphiris (Eds.), Human-computer interaction
—INTERACT 2019 (pp. 67–87). Springer International Publishing.
48. Schneegass, S., Saad, A., Heger, R., Delgado, S., Poguntke, R., & Alt, F. (2022). An
investigation of shoulder surfing attacks on touch-based unlock events. In Proceedings of
the 24th International Conference on Human-Computer Interaction with Mobile Devices and
Services, MobileHCI ’22. Association for Computing Machinery. To Appear.
49. Schneegass, S., Steimle, F., Bulling, A., Alt, F., & Schmidt, A. (2014). SmudgeSafe: Geometric
image transformations for smudge-resistant user authentication. In Proceedings of the 2014
ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp ’14
(pp. 775–786). Association for Computing Machinery.
50. Schubert, T. W. (2003). The sense of presence in virtual environments: A three-component
scale measuring spatial presence, involvement, and realness. Zeitschrift für Medienpsychologie,
15(2), 69–71.
51. Schwind, V., Knierim, P., Haas, N., & Henze, N. (2019). Using presence questionnaires in
virtual reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing
Systems, volume 2019 of CHI ’19 (pp. 1–12). Association for Computing Machinery.
52. Shin, H., Sim, S., Kwon, H., Hwang, S., & Lee, Y. (2022). A new smart smudge attack using
CNN. International Journal of Information Security, 21(1), 25–36.
53. Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin,
133(5), 859.
54. von Zezschwitz, E., De Luca, A., Janssen, P., & Hussmann, H. (2015). Easy to draw, but
hard to trace? On the observability of grid-based (un)lock patterns. In Proceedings of the 33rd
Annual ACM Conference on Human Factors in Computing Systems, CHI ’15 (pp. 2339–2342).
Association for Computing Machinery.
55. Watanabe, K., Higuchi, F., Inami, M., & Igarashi, T. (2012). CursorCamouflage: Multiple
dummy cursors as a defense against shoulder surfing. In SIGGRAPH Asia 2012 Emerging
Technologies, SA ’12 (pp. 1–2). Association for Computing Machinery.
218 A. Saad et al.
56. Wiese, O., & Roth, V. (2015). Pitfalls of shoulder surfing studies. In In NDSS Workshop on
Usable Security 2015 (USEC’15) ( pp. 1–6). Internet Society.
57. Wiese, O., & Roth, V. (2016). See you next time: A model for modern shoulder surfers.
In Proceedings of the 18th International Conference on Human-Computer Interaction with
Mobile Devices and Services, MobileHCI ’16 (pp. 453–464). Association for Computing
Machinery.
58. Winkler, C., Gugenheimer, J., De Luca, A., Haas, G., Speidel, P., Dobbelstein, D., & Rukzio,
E. (2015). Glass unlock: Enhancing security of smartphone unlocking through leveraging a pri-
vate near-eye display. In Proceedings of the 33rd Annual ACM Conference on Human Factors
in Computing Systems, CHI ’15 (pp. 1407–1410). Association for Computing Machinery.
59. Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence
questionnaire. Presence: Teleoperators and Virtual Environments, 7(3), 225–240.
60. Xu, Y., Heinly, J., White, A. M., Monrose, F., & Frahm, J.-M. (2013). Seeing double:
Reconstructing obscured typed input from repeated compromising reflections. In Proceedings
of the 2013 ACM SIGSAC Conference on Computer & Communications Security, CCS ’13 (pp.
1063–1074). Association for Computing Machinery.
61. Ye, G., Tang, Z., Fang, D., Chen, X., Wolff, W., Aviv, A. J., & Wang, Z. (2018). A video-based
attack for Android pattern lock. ACM Transactions on Privacy and Security, 21(4), 1–31.
62. Zhou, H., Ferreira, V., Alves, T., Hawkey, K., & Reilly, D. (2015). Somebody is peeking!
A proximity and privacy aware tablet interface. In Proceedings of the 33rd Annual ACM
Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA ’15 (pp.
1971–1976). Association for Computing Machinery.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing,
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate
credit to the original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this chapter are included in the chapter’s Creative
Commons license, unless indicated otherwise in a credit line to the material. If material is not
included in the chapter’s Creative Commons license and your intended use is not permitted by
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from
the copyright holder.
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
The General Data Protection Regulation (GDPR) was implemented in the European Union and European Economic Area in May 2018. The GDPR aims to strengthen consumers’ rights to data privacy in the wake of technological developments like big data and artificial intelligence. This was a hot topic for stakeholders, such as lawyers, companies and consumers, prior to the GDPR’s implementation. This paper investigates to what extent consumers are concerned about information privacy issues following the implementation of the GDPR. We present findings from an online survey conducted during spring 2019 among 327 Norwegian consumers, as well as findings from a survey conducted immediately prior to the implementation of the GDPR in spring 2018. We draw the following conclusions: (1) consumers gained significant knowledge about their information privacy from the GDPR, but felt relatively little need to execute their enhanced rights; (2) about 50% of respondents believed themselves to have control over their data, while almost 40% stated that they had no control about their personal data; and (3) consumers largely trusted companies to manage their personal data. These insights are of interest to both academia and to industries that deal with personal data.
Poster
Full-text available
Shoulder surfing is a prevailing threat when accessing smartphone information at different locations. Prior work has proposed numerous mechanisms to combat the threat, however, when and what mechanism to use while maintaining appreciable user experience and usability remains a challenge. Further, the subjective interpretation of sensitive content adds to the challenge of protecting users’ privacy and security. In this poster, we present preliminary findings on what users perceive as sensitive information in the context of shoulder surfing from an online survey with N= 40 participants. We found that the need for the protection mechanism varies with the context of use. Users consider location and relationship with the observer when hiding content from unconsented observations. Based on the findings, we propose a typology of perceived sensitive content considering social aspects in shoulder surfing scenarios. Our typology can be used as a baseline for designing personalized shoulder surfing protection mechanisms.
Article
Full-text available
This paper contributes to our understanding of user-centered attacks on smartphones. In particular, we investigate the likelihood of so-called shoulder surfing attacks during touch-based unlock events and provide insights into users’ views and perceptions. To do so, we ran a two-week in-the-wild study (N=12) in which we recorded images with a 180-degree field of view lens that was mounted on the smartphone’s front-facing camera. In addition, we collected contextual information and allowed participants to assess the situation. We found that only a small fraction of shoulder surfing incidents that occur during authentication are actually perceived as threatening. Furthermore, our findings suggest that our notions of (un)safe places need to be rethought. Our work is complemented by a discussion of implications for future user-centered attack-aware systems. This work can serve as a basis for usable security researchers to better design systems against user-centered attacks.
Article
Full-text available
One approach to mitigate shoulder surfing attacks on mobile devices is to detect the presence of a bystander using the phone’s front-facing camera. However, a person’s face in the camera’s field of view does not always indicate an attack. To overcome this limitation, in a novel data collection study (N=16), we analysed the influence of three viewing angles and four distances on the success of shoulder surfing attacks. In contrast to prior works that mainly focused on user authentication, we investigated three common types of content susceptible to shoulder surfing: text, photos, and PIN authentications. We show that the vulnerability of text and photos depends on the observer’s location relative to the device, while PIN authentications are vulnerable independent of the observation location. We then present PrivacyScout – a novel method that predicts the shoulder-surfing risk based on visual features extracted from the observer’s face as captured by the front-facing camera. Finally, evaluations from our data collection study demonstrate our method’s feasibility to assess the risk of a shoulder surfing attack more accurately.
Article
The SARS-CoV-2 pandemic is a pressing societal issue today. The German government promotes a contract tracing app named Corona-Warn-App (CWA), aiming to change citizens' health behaviors during the pandemic by raising awareness about potential infections and enable infection chain tracking. Technical implementations, citizens' perceptions, and public debates around apps differ between countries, e. g., in Germany there has been a huge discussion on potential privacy issues of the app. Thus, we analyze effects of privacy concerns regarding the CWA, perceived CWA benefits, and trust in the German healthcare system to answer why citizens use the CWA. In our initial conference publication at ICT Systems Security and Privacy Protection - 37th IFIP TC 11 International Conference, SEC 2022, we used a sample with 1752 actual users and non-users of the CWA and and support for the privacy calculus theory, i. e., individuals weigh privacy concerns and benefits in their use decision. Thus, citizens privacy perceptions about health technologies (e. g., shaped by public debates) are crucial as they can hinder adoption and negatively affect future fights against pandemics. In this special issue, we adapt our previous work by conducting a second survey 10 months after our initial study with the same pool of participants (830 participants from the first study participated in the second survey). The goal of this longitudinal study is to assess changes in the perceptions of users and non-users over time and to evaluate the influence of the significantly lower hospitalization and death rates on the use behavior which we could observe during the second survey. Our results show that the privacy calculus is relatively stable over time. The only relationship which significantly changes over time is the effect of privacy concerns on the use behavior which significantly decreases over time, i. e., privacy concerns have a lower negative effect one the CWA use indicating that it did not play such an important role in the use decision at a later point in time in the pandemic. We contribute to the literature by introducing one of the rare longitudinal analyses in the literature focusing on the privacy calculus and changes over time in the relevant constructs as well as the relationships between the calculus constructs and target variables (in our case use behavior of a contact tracing app). We can see that the explanatory power of the privacy calculus model is relatively stable over time even if strong externalities might affect individual perceptions related to the model.
Conference Paper
The increasing number of smart devices installed in our homes poses privacy risks for inhabitants and visitors. However, individuals face difficulties counteracting privacy intrusions due to missing controls, incorrect mental models, and limitations in their level of expertise. We present PriKey, a concept for device-independent and easy-to-use tangible smart home privacy mechanisms. PriKey is the key to privacy protection: it supports users in taking control over their privacy through meaningful, tangible interactions. Using a Wizard-of-Oz prototype, we explored users’ perceptions regarding PriKey (N = 16). We then compared PriKey to an equivalent smartphone app (N = 32), focusing on visitors. Participants perceived PriKey as engaging, intuitive, and benevolent. Their privacy considerations were based on personal and contextual factors. While most participants preferred the smartphone app, others clearly favored PriKey. Our results indicate that tangible privacy is a noteworthy approach for future smart home privacy mechanisms.