Conference PaperPDF Available

The Curious Case of the Curious Case: Detecting Touchscreen Events Using a Smartphone Protective Case

Authors:
The Curious Case of the Curious Case:
Detecting touchscreen events using a smartphone protective case
The Anonymous Authors
Abstract—Security-conscious users are very careful with soft-
ware they allow their phone to run. They are much less
careful with the choices they make regarding accessories such
as headphones or chargers and only few, if any, care about
cyber security threats coming from the phone’s protective
case. We show how a malicious smartphone protective case
can be used to detect and monitor the victim’s interaction
with the phone’s touchscreen, opening the door to keylogger-
like attacks, threatening the user’s security and privacy. This
feat is achieved by implementing a hidden capacitive sensing
mechanism inside the case. Our attack is both sensitive enough
to track the user’s finger location across the screen, and
simple and cheap enough to be mass-produced and deployed en
masse. We discuss the theoretical principles behind this attack,
present a preliminary proof-of-concept, and discuss potential
countermeasures and mitigations.
Index Terms—touchscreen leak, security, privacy, smartphone.
1. Introduction
Personal mobile devices are widely popular and are now
used to perform sensitive tasks such as banking, sending and
receiving work-related emails, and even connecting remotely
to servers. In each of these tasks the user enters private data
into the device using the touchscreen, either for the purpose
of authentication or as part of the data exchange. Adversaries
who are interested in obtaining touch event information
often use malicious software.
Many users take protective measures, such as using
anti-malware or following cautious behavior, for protecting
their private information. Organizations define strict security
policy that is applied by installing dedicated software and/or
hardware components on the employees phones. None of
these measures, however, consider the phone protective case
as a security risk. Thus, for higher-value targets, malicious
adversaries may decide to make use of custom hardware
devices, commonly called bugs or implants.
As discussed by Farshteindiker et al. in [1], a malicious
implant has three main functional requirements: first, it must
be able to collect data from its victim; next, it must be
able to exfiltrate the data to the attacker; finally, the implant
requires some sort of power supply to power its computation
and communication functions. Farshteindiker et al. suggest
a method by which an implant in close proximity to the
phone can use the phone itself to exfiltrate data, and also
discuss several potential options for supplying power to
such a device. Our paper focuses on the data collection
functionality of the implant. In particular, we present a
method for remotely detecting touch screen events using
a malicious smartphone protective case, which is cheap and
simple enough to be mass-produced.
In order to get the malicious protective case onto the
user’s phone, the attacker can use a social engineering attack
in which the case is distributed or given as a gift or souvenir
(e.g., at a trade show or scientific conference). For mass
information theft, the implant can be injected into the supply
chain using various supply-chain interdiction methods [2].
In this study, we focus on protective cases as an attack
platform, because they are widely used, are almost always
attached to the victim’s phone (in contrast to chargers or
headphones), and surround the phone on all sides (a fact that
aids in detection of touch events). Using a case has the addi-
tional advantage of working on any device, regardless of its
operating system or internal structure. Since the malicious
case is completely separated from the phone’s hardware and
software, it does not require a dedicated application to run
on the phone, and it work in a ”plug and play” manner.
Contribution. We design and evaluate a capacitive sen-
sor which can be implanted inside a phone’s protective case
and is capable of detecting the motion of a user’s finger on a
smartphone’s touch screen with a high precision. We discuss
the hardware and software requirements of our sensor and
show how it can be used to recover a phone’s secret unlock
pattern. After the motion data is analyzed, it can be stored
or transmitted to the attacker using one of the methods
described in prior research [1].
2. Capacitive Sensing
The touchscreens of nearly all modern smartphones and
tablets use capacitive sensing to detect touch on the screen.
There are two main methods of capacitive sensing: surface
capacitance and projected capacitance.
In the surface capacitance sensing method, four elec-
trodes are placed on the four edges of the screen and they are
connected to a conductive layer overlay the whole screen.
When the user’s finger touches the screen’s conductive sur-
face, a voltage drop measured by these electrodes changes
relative to the finger’s distance from the electrodes.
In the projected capacitance sensing method, a grid of
equidistant electrodes is placed under the screen glass. Each
adjacent pair of electrodes then forms a capacitor with a
fixed capacitance, which is measurable by applying voltage
to one of the electrodes and measuring how long it takes for
the voltage on the other electrode to reach a certain value.
Placing an conductive object, such as a finger, in the mag-
netic field emitted by the electrodes, increases the amount
of charge that can be stored in the electrodes and hence
increases the capacitance. Consequently, the time required
to charge the capacitor increases as the finger gets closer to
the electrodes. In contrast to the surface capacitance method,
this method requires an additional layer of signal processing
and noise reduction to derive the finger’s exact coordinates.
In our implementation, we use the projected capacitive
sensing method due to its simplicity and ability to detect
the finger without actual contact with the sensor.
3. Attack Description
The main idea behind the discussed attack is obtaining
the touch position on a touchscreen using a simple looking
protective case. In order for the sensor to be implementable
in a typical phone protective case, it needs to be located
on the edges of the phone. In addition to capacitive sensing,
there are a few alternative methods that can be used for touch
sensing using a sensor that circles the screen. Those methods
include Infrared (IR) touch sensing or surface acoustic wave
(SAW) touch sensing. Neither of these methods is suited for
the task, since the IR sensor needs relatively large bezels to
fit the LEDs, and the SAW touch will only work with very
good contact between the sensor and the screen, something
that a plastic case can not provide.
As mentioned, our implementation uses the projected
capacitive sensing method. Our setup consists of four elec-
trodes, acting as receivers, and an additional electrode that
serves as the transmitter. Not to be confused with the four
electrodes of the surface capacitance, our implementation
is not connected to conductive layer on the screen. As seen
in Figure 1 (blue bars), the four electrodes are placed on
the inner part of the case, one on each of the four sides.
The transmitting electrode sits on the back of the phone.
Traditional projected capacitive touchscreens use an array
of electrodes that are placed very close to each other. This
arrangement can only detect touch events in very short dis-
tances, usually covering just the thickness of the glass so it
can detect touch. In our implementation, capacitance is used
differently than regular projected capacitance; since it has
a wider gap between electrodes, it can sense and measure
a greater distance of the finger from the four electrodes. In
the following demonstration we attempt to obtain a phone’s
unlock pattern, commonly used in Android smartphones.
3.1. Evaluation Setup
We designed and conducted an experiment to demon-
strate and evaluate the effectiveness of the studied attack.
The hardware setup of our experiment is presented in Fig-
ure 3. We concentrated on keeping the setup as cheap and
simple as possible, both in terms of the choice of compo-
nents and the choice of construction methods. We used a
Samsung Galaxy S5, and all of the tests were conducted
while the phone was turned on and placed flat on a desk,
with one finger touching the screen.
The five electrodes composing the touch sensor were
made of ordinary aluminum foil. Four electrodes, placed on
Micro processor
Figure 1: Electrode placement relative to the phone
Figure 2: Experimental setup
the sides of the phone, are cut into a narrow rectangular
shape (two 14cm long and two 7cm long) that runs the
length of the sides, top and bottom of the phone. The
electrode placed on the back of the phone is also a rectangle
(10cm×3cm), sized so that it can sit at the center of the back
of the phone approximately 2cm from the phone edges. All
of the electrodes are taped to the phone, with an isolation
layer between them and the phone frame. A striped wire is
taped to each electrode to make a conductive connection on
one end, and they are connected to the MCU (microcon-
troller unit) pins on the other end. Near the MCU, between
the input and each of the outputs, we have connected a 1M
ohm resistor (for a total of four resistors).
The MCU used is an Arduino Nano board based on an
ATMEL-MEGA328p chip. It uses five ports to connect to
the five electrodes. The sampling rate of the sensor was
set to 200Hz. The MCU executes a code that uses the
open-source capacitive sensor library [3]. After performing
signal processing, the MCU sends the data to a PC using
a serial connection over USB. After each trial we restarted
the system to eliminate changes in the test environment.
In a real world implementation the electrodes will be
attached to the protective case on the inner side covered
with an isolation layer of the same color as the protective
case itself. The electronic circuit implementing the sensor
with the simple signal processing and RF power source can
be located on phone cover. That include wires, electronic
components, MCU and a battery all can be thin enough
to fit in a phone case. The electromagnetic noise that this
circuit will emit is expected to be negligible compared to the
electromagnetic emission of the phone and thus, the device
itself will not interfere with the touch sensor. The device
can exfiltrate the collected data in a concealed way using
the methods proposed in [1].
The attacker’s device analyzes the data collected from
the four sensing electrodes and converts it to X and Y
coordinates on the two-dimensional screen surface. This data
can then be further analyzed to extract keyboard presses,
lock screen PINs, or patterns.
Victim’s Device
Sensor
Basic Signal
Processing
MicroController
Raw Data
Analysis
בData
Exporting
Attacker’s Device
Figure 3: Design flow
3.2. Evaluation Results
During initial experiments we noticed an important,
phenomenon. As the user’s finger gets closer to the screen,
the capacitance reading increases as expected. However, at
the moment the finger touched the screen itself, a significant
change in the reading is registered. This important finding
allows us to differentiate between a finger hovering over
the screen and an actual touch event. We suspect that this
phenomenon is caused by the conductive layer located just
below the glass of the screen, the same layer that is used
for the native touchscreen capabilities of the phone. We note
again that this measurement was obtained by sensors that
surround the phone, but do not touch the user’s finger or
the screen’s surface.
The three examples illustrated in Figure 4 show that the
patterns drawn on the phone’s screen can be reconstructed
from measurements obtained by the external touch sensor.
We evaluated the implementation by drawing on the
screen and capturing five different patterns (the letters L, O,
Z, U and S); each pattern was drawn 30 times. We applied
a simple pattern recognition algorithm1that calculates the
path distance of a sample (i.e., captured pattern) and patterns
provided by a preconfigured dictionary. As presented in
Table 1 the naive algorithm is able to identify the letters with
an average accuracy of 75%. Thus, it should be possible to
reconstruct coarse touch events such as the phone’s secret
unlock pattern. Additional processing would be required
1. https://mccormick.cx/news/entries/gesture-recognition
(a) ’Z’ shaped pat-
tern
(b) ’L’ shaped
pattern
(c) ’U’ shaped pat-
tern
Figure 4: Sensor outputs for 3 patterns drawn on the screen
Letter Success Rate
L 76.6%
O 73.3%
Z 63.6%
U 80.0%
S 93.3%
Table 1: Success rate for five patterns.
before we are able to reconstruct fine-grained touch events
such as keyboard typing.
Environmental effects had an impact on our readings.
When the user’s hand was placed under the phone, our
ability to obtain readings decreased. Although the changes in
capacitance due to touch events were still noticeable in the
raw sensor outputs, our test setup was unable to convert the
measurements into X-Y coordinates under these conditions.
In a full attack setup, the attacker would probably have to
use additional technics to eliminate the Parasitic capacitive
coupling, and apply some basic signal processing to elimi-
nate the effects of this activity on the final measurements.
The readings were also affected when we connected a
charger to the phone. In that particular case, the bottom
electrode’s sensitivity dropped, probably because the phone
was connected to the charger’s ground plane. However, all of
the other electrodes continued obtaining standard readings,
still making the attack possible.
4. Discussion
We demonstrate a technology that can be embedded in
a simple looking phone protective case and can be used to
detect the finger pressing position on a screen. This attack
vector is different from other methods, as it requires minimal
involvement of the attacker in the physical environment of
the victim, and none in the software environment. This
attack is cheap and easy enough to mass-produce. It is
possible to implement the attack on a large scale by inserting
the touch logging device in the supply chain of phone
protective cases.
Currently, users or organizations don’t think of a smart-
phone protective case as an item that poses a security
risk, and most efforts to secure devices focus on protecting
communication and software. Simple protective cases are
routinely given as gifts and sold at stores and on the Internet.
Most, if not all, users would not hesitate to put on a
protective case, even if the source of the case was unknown
or considered untrustworthy.
5. Countermeasures
In order to be successful, the attack vector relies on on
the ability to get the user to put the malicious smartphone
protective case on his/her device. In order to achieve this
feat the attacker participate in an official event and give the
protective cases away as a souvenir or prize. Someone who
purchases a protective case from a store faces the same risk,
in the case of supply chain insertion.
A good way to eliminate this type of threat is to use
a transparent protective case. Essential components of the
attack mechanism cannot be hidden in a transparent case.
In contrast, there are some phone protective case materials
and features that are more suited to the attack mechanism;
therefore, a case made of a stiff, thick, or opaque material is
more likely to be used in this type of attack. A case with an
external battery is an excellent candidate as they also have
power source that can power the system.
In order to eliminate any suspicions about a given phone
protective case, it is possible to use an x-ray scanner or metal
detector or some other mechanism that can identify system
components. Obviously, such tests are not relevant for a case
with an external battery which contains metal and electric
components.
It is also possible to produce phones that are less sus-
ceptible to this kind of attack. A non-conductive stylus such
as those used with resistive touch screens can be effective
against this attack. Phones screens that don’t use the type of
touch screen technologies that have a conductive layer near
the screen (e.g., IR and SAW) will not result in an immediate
change at the sensor reading when a finger touches the
screen. The lack of the touch indication capability makes
it more difficult to distinguish between a hovering finger
and a touching finger. Although possibly helpful in the face
of the discussed attack, it is unlikely that manufacturers will
abandon this type of touchscreen.
Finally, the most effective countermeasure is user aware-
ness; the user must be suspicious about any phone accessory
coming from unknown or Untrustworthy sources.
6. Related Works
Previous work has shown that touchscreen events can
be gathered using malicious software running on the smart-
phone. This is the case even if the attacker cannot di-
rectly access the phone’s touchscreen API, and only has
access to a subset of its other sensors [4]–[6]. The use
of anti-virus software, strengthening the operating system,
and simply exercising user caution can serve as effective
defensive measures against such software-based attacks. In
2007, Sekiguchi demonstrated an approach for detecting
touch screen events based on analysis of the electromagnetic
noise emitted by the touchscreen [7]. This method requires
expensive equipment, and is difficult for the attacker to
carry out in an adversarial situation. In 2015, Ali et al.
showed how the formation and direction of a user’s hands
can be detected by their effect on Wi-Fi signals [8]. Their
attack assumed that the victim is positioned between two
cooperating Wi-Fi transceivers, an assumption which is less
practical in the context of a hardware implant. Low-tech
measures such as shoulder surfing [9] can also be used to
obtain touch screen data.
7. Conclusions and Future Work
We investigated the use of a malicious smartphone pro-
tective case as a side-channel to infer patterns drawn on
a smartphone touchscreen. We observed that a case with
dedicated sensors can determine the finger location on the
touchscreen. We developed a proof-of-concept that uses the
sensor readings and processing to infer the finger movement
pattern on the screen. We have demonstrated that smart-
phone protective case must be considered a security threat
as it may serve as a side channel from which confidential
information can be leaked from a smartphone. In future work
we intend to deeper investigate the sensor design. We will
evaluate end-to-end attack that is able to detect soft keyboard
events and and leak the events to an external server and
propose capacitive touch screen design that may be used to
detect rogue capacitive sensors.
References
[1] A. G. B. Farshteindiker, N. Hasidim and Y. Oren, “How to phone home
with someone else’s phone: Information exfiltration using intentional
sound noise on gyroscopic sensors,” USENIX Association, 2016.
[2] J. Appelbaum, A. Gibson, C. Guarnieri, A. M¨
uller-Maguhn, L. Poitras,
M. Rosenbach, L. Ryge, H. Schmundt, and M. Sontheimer, “The digital
arms race: NSA preps america for future battle,” Der Spiegel, vol. 1,
no. 17, Jan 2015.
[3] PaulStoffregen, “Capacitivesensor,” web, 2014,
github.com/PaulStoffregen/CapacitiveSensor.
[4] L. Cai, S. Machiraju, and H. Chen, “Defending against sensor-sniffing
attacks on mobile phones,” in Proceedings of the 1st ACM SIGCOMM
Workshop on Networking, Systems, and Applications for Mobile Hand-
helds, MobiHeld 2009, 2009.
[5] L. Cai and H. Chen, “Touchlogger: Inferring keystrokes on touch
screen from smartphone motion,” in 6th USENIX Workshop on Hot
Topics in Security, HotSec’11, San Francisco, CA, USA, August 9,
2011.
[6] R. Schlegel, K. Zhang, X. Zhou, M. Intwala, A. Kapadia, and X. Wang,
“Soundcomber: A stealthy and context-aware sound trojan for smart-
phones,” in Proceedings of the Network and Distributed System Secu-
rity Symposium (NDSS), 2011.
[7] H. Sekiguchi, “Novel information leakage threat for input operations
on touch screen monitors caused by electromagnetic noise and its
countermeasure method,” Progress In Electromagnetics Research B,
vol. 36, no. 36, pp. 399–419, 2012.
[8] K. Ali, A. X. Liu, W. Wang, and M. Shahzad, “Keystroke recognition
using wifi signals,” in Proceedings of the 21st Annual International
Conference on Mobile Computing and Networking, MobiCom 2015,
2015.
[9] A. H. Lashkari, S. Farmand, O. B. Zakaria, and R. Saleh, “Shoul-
der surfing attack in graphical password authentication,” CoRR, vol.
abs/0912.0951, 2009.
... Other examples of physical channels exploited by malicious entities are recovery partition (i.e., recovery mode) (Vidas et al., 2011), malicious chargers (Lau et al., 2013), protection cases (Gluck et al., 2017), and replacement screens (Shwartz et al., 2017). The users' security awareness of physical channels and the danger that exists in connecting to untrusted components can protect them from exposing their device to "physical" attacks. ...
Article
The popularity of smartphones, coupled with the amount of valuable and private information they hold, make them attractive to attackers interested in exploiting the devices to harvest sensitive information. Exploiting human vulnerabilities (i.e., social engineering) is an approach widely used to achieve this goal. Improving the security awareness of users is an effective method for mitigating social engineering attacks. However, while in the domain of personal computers (PCs) the security awareness of users is relatively high, previous studies have shown that for the mobile platform, the security awareness level is significantly lower. The skills required from a mobile user to interact safely with his/her smartphone are different from those that are required for safe and responsible PC use. Therefore, the awareness of mobile users to security risks is an important aspect of information security. An essential and challenging requirement of assessing security awareness is the definition of measureable criteria for a security aware user. In this paper, we present a hierarchical taxonomy for security awareness, specifically designed for mobile device users. The taxonomy defines a set of measurable criteria that are categorized according to different technological focus areas (e.g., applications and browsing) and within the context of psychological dimensions (e.g., knowledge, attitude, and behavior). We demonstrate the applicability of the proposed taxonomy by introducing an expert-based procedure for deriving mobile security awareness models for different attack classes (each class is an aggregation of social engineering attacks that exploit a similar set of human vulnerabilities). Each model reflects the contribution (weight) of each criteria to the mitigation of the corresponding attack class. Application of the proposed procedure, based on the input of 17 security experts, to derive mobile security awareness models of four different attack classes, confirms that the skills required from a smartphone user to mitigate an attack are different for different attack classes.
Conference Paper
Full-text available
We explore the threat of smartphone malware with ac-cess to on-board sensors, which opens new avenues for il-licit collection of private information. While existing work shows that such "sensory malware" can convey raw sen-sor data (e.g., video and audio) to a remote server, these approaches lack stealthiness, incur significant communica-tion and computation overhead during data transmission and processing, and can easily be defeated by existing pro-tections like denying installation of applications with ac-cess to both sensitive sensors and the network. We present Soundcomber, a Trojan with few and innocuous permis-sions, that can extract a small amount of targeted private information from the audio sensor of the phone. Using targeted profiles for context-aware analysis, Soundcomber intelligently "pulls out" sensitive data such as credit card and PIN numbers from both tone-and speech-based inter-action with phone menu systems. Soundcomber performs efficient, stealthy local extraction, thereby greatly reducing the communication cost for delivering stolen data. Sound-comber automatically infers the destination phone number by analyzing audio, circumvents known security defenses, and conveys information remotely without direct network access. We also design and implement a defensive architec-ture that foils Soundcomber, identify new covert channels specific to smartphones, and provide a video demonstration of Soundcomber.
Article
Full-text available
Information and computer security is supported largely by passwords which are the principle part of the authentication process. The most common computer authentication method is to use alphanumerical username and password which has significant drawbacks. To overcome the vulnerabilities of traditional methods, visual or graphical password schemes have been developed as possible alternative solutions to text-based scheme. A potential drawback of graphical password schemes is that they are more vulnerable to shoulder surfing than conventional alphanumeric text passwords. When users input their passwords in a public place, they may be at risk of attackers stealing their password. An attacker can capture a password by direct observation or by recording the individual’s authentication session. This is referred to as shouldersurfing and is a known risk, of special concern when authenticating in public places. In this paper we will present a survey on graphical password schemes from 2005 till 2009 which are proposed to be resistant against shoulder surfing attacks.
Conference Paper
Keystroke privacy is critical for ensuring the security of computer systems and the privacy of human users as what being typed could be passwords or privacy sensitive information. In this paper, we show for the first time that WiFi signals can also be exploited to recognize keystrokes. The intuition is that while typing a certain key, the hands and fingers of a user move in a unique formation and direction and thus generate a unique pattern in the time-series of Channel State Information (CSI) values, which we call CSI-waveform for that key. In this paper, we propose a WiFi signal based keystroke recognition system called WiKey. WiKey consists of two Commercial Off-The-Shelf (COTS) WiFi devices, a sender (such as a router) and a receiver (such as a laptop). The sender continuously emits signals and the receiver continuously receives signals. When a human subject types on a keyboard, WiKey recognizes the typed keys based on how the CSI values at the WiFi signal receiver end. We implemented the WiKey system using a TP-Link TL-WR1043ND WiFi router and a Lenovo X200 laptop. WiKey achieves more than 97.5% detection rate for detecting the keystroke and 96.4% recognition accuracy for classifying single keys. In real-world experiments, WiKey can recognize keystrokes in a continuously typed sentence with an accuracy of 93.5%.
Article
Information leakage of general input operations using button images in graphical user interface on touch screen monitors was experimentally investigated from images reconstructed by receiving the electromagnetic noise. In the experimental investigations for input operations of a personal identification number, it was confirmed that when a button image was touched, the touched button image can be identified from the reconstructed button images. This kind of information leakage has originated the fact that the touched button image has changed the color for informing the operator which button image was touched. From the elucidation of the image reconstruction mechanism, it was found that the information leakage has been caused by the magnitude of the emitted signal that results from the analog voltage differences of the RGB signals between neighboring pixels on the monitor. Therefore, a countermeasure method was proposed from the viewpoint of the combination of the colors of the button images and of the background or of the numerals in the button images. The countermeasure method was then applied to the previous input operations of a personal identification number. From the experimental results for the countermeasure method, it was confirmed that the touched button image cannot be identified from the reconstructed button image. As a result, the proposal countermeasure method can prevent effectively the information leakage of input operations on touch screen monitors due to the electromagnetic noise.
Conference Paper
Attacks that use side channels, such as sound and electromagnetic emanation, to infer keystrokes on physical keyboards are ineffective on smartphones without physical keyboards. We describe a new side channel, motion, on touch screen smartphones with only soft keyboards. Since typing on different locations on the screen causes different vibrations, motion data can be used to infer the keys being typed. To demonstrate this attack, we developed TouchLogger, an Android application that extracts features from device orientation data to infer keystrokes. TouchLogger correctly inferred more than 70% of the keys typed on a number-only soft keyboard on a smartphone. We hope to raise the awareness of motion as a significant side channel that may leak confidential data.
Conference Paper
Modern mobile phones possess three types of capabilities: computing, communication, and sensing. While these capa- bilities enable a variety of novel applications, they also raise serious privacy concerns. We explore the vulnerability where attackers snoop on users by sniffing on their mobile phone sensors, such as the microphone, camera, and GPS receiver. We show that current mobile phone platforms inadequately protect their users from this threat. To provide better pri- vacy for mobile phone users, we analyze desirable uses of these sensors and discuss the properties of good privacy pro- tection solutions. Then, we propose a general framework for such solutions and discuss various possible approaches to implement the framework's components.
The digital arms race: NSA preps america for future battle
  • J Appelbaum
  • A Gibson
  • C Guarnieri
  • A Müller-Maguhn
  • L Poitras
  • M Rosenbach
  • L Ryge
  • H Schmundt
  • M Sontheimer
J. Appelbaum, A. Gibson, C. Guarnieri, A. Müller-Maguhn, L. Poitras, M. Rosenbach, L. Ryge, H. Schmundt, and M. Sontheimer, "The digital arms race: NSA preps america for future battle," Der Spiegel, vol. 1, no. 17, Jan 2015.