ChapterPDF Available

Is My Phone Listening in? On the Feasibility and Detectability of Mobile Eavesdropping

Authors:

Abstract and Figures

Besides various other privacy concerns with mobile devices, many people suspect their smartphones to be secretly eavesdropping on them. In particular, a large number of reports has emerged in recent years claiming that private conversations conducted in the presence of smartphones seemingly resulted in targeted online advertisements. These rumors have not only attracted media attention, but also the attention of regulatory authorities. With regard to explaining the phenomenon, opinions are divided both in public debate and in research. While one side dismisses the eavesdropping suspicions as unrealistic or even paranoid, many others are fully convinced of the allegations or at least consider them plausible. To help structure the ongoing controversy and dispel misconceptions that may have arisen, this paper provides a holistic overview of the issue, reviewing and analyzing existing arguments and explanatory approaches from both sides. Based on previous research and our own analysis, we challenge the widespread assumption that the spying fears have already been disproved. While confirming a lack of empirical evidence, we cannot rule out the possibility of sophisticated large-scale eavesdropping attacks being successful and remaining undetected. Taking into account existing access control mechanisms, detection methods, and other technical aspects, we point out remaining vulnerabilities and research gaps.
Content may be subject to copyright.
Is My Phone Listening in? On the Feasibility
and Detectability of Mobile Eavesdropping
Jacob Leon Kröger
1,2(&)
and Philip Raschke
1
1
Technische Universität Berlin, Berlin, Germany
{kroeger,philip.raschke}@tu-berlin.de
2
Weizenbaum Institute for the Networked Society, Berlin, Germany
Abstract. Besides various other privacy concerns with mobile devices, many
people suspect their smartphones to be secretly eavesdropping on them. In
particular, a large number of reports has emerged in recent years claiming that
private conversations conducted in the presence of smartphones seemingly
resulted in targeted online advertisements. These rumors have not only attracted
media attention, but also the attention of regulatory authorities. With regard to
explaining the phenomenon, opinions are divided both in public debate and in
research. While one side dismisses the eavesdropping suspicions as unrealistic
or even paranoid, many others are fully convinced of the allegations or at least
consider them plausible. To help structure the ongoing controversy and dispel
misconceptions that may have arisen, this paper provides a holistic overview of
the issue, reviewing and analyzing existing arguments and explanatory
approaches from both sides. Based on previous research and our own analysis,
we challenge the widespread assumption that the spying fears have already been
disproved. While conrming a lack of empirical evidence, we cannot rule out
the possibility of sophisticated large-scale eavesdropping attacks being suc-
cessful and remaining undetected. Taking into account existing access control
mechanisms, detection methods, and other technical aspects, we point out
remaining vulnerabilities and research gaps.
Keywords: Privacy Smartphone Eavesdropping Spying Listening
Microphone Conversation Advertisement
1 Introduction
Smartphones are powerful tools that make our lives easier in many ways. Since they are
equipped with a variety of sensors, store large amounts of personal data and are carried
throughout the day by many people, including in highly intimate places and situations,
they also raise various privacy concerns.
One widespread fear is that smartphones could be turned into remote bugging
devices. For years, countless reports have been circulating on the Internet from people
who claim that things they talked about within earshot of their phone later appeared in
targeted online advertisements, leading many to believe that their private conversations
must have been secretly recorded and analyzed.
©The Author(s) 2019
S. N. Foley (Ed.): DBSec 2019, LNCS 11559, pp. 102120, 2019.
https://doi.org/10.1007/978-3-030-22479-0_6
The reported suspicious ads range across many product and service categories,
including clothing, consumer electronics, foods and beverages, cars, medicines, holi-
day destinations, sports equipment, pet care products, cosmetics, and home appliances
and while some of these ads were described as matching an overall discussion topic,
others allegedly promoted a brand or even a very specic product mentioned in a
preceding face-to-face conversation [6,12]. Some people claim to have experienced the
phenomenon frequently and that they have successfully reproduced it in private
experiments. Interestingly, many of the purported witnesses emphasize that the
advertised product or service seems not related to places they have visited, terms they
have searched for online, or things they have mentioned in text messages, emails or
social media [6,40]. Furthermore, some reports explicitly rate it as unlikely that the
respective advertisements were selected by conventional targeting algorithms, as they
lay notably outside the range of advertising normally received and did sometimes not
even appear to match the persons consumer prole (e.g. in terms of interests, activities,
age, gender, or relationship status) [6,41].
Numerous popular media outlets have reported on these alleged eavesdropping
attacks [3]. In a Forbes article, for instance, the US-based market research company
Forrester reports that at least 20 employees in its own workforce have experienced the
phenomenon for themselves [40]. The same holds true for one in ve Australians,
according to a recent survey [38]. Even the US House Committee on Energy and
Commerce has started to investigate the issue by sending letters to Google and Apple
inquiring about the ways in which iOS and Android devices record private conver-
sations [77].
Many commentators, including tech bloggers, researchers and business leaders, on
the other hand, view the fear that private companies could target their ads based on
eavesdropped conversations as baseless and paranoid. The reputational risk, it is
argued, would be far too high to make this a viable option [76]. With regard to CPU,
battery and data storage limitations, former Facebook product manager Antonio García
Martínez even considers the alleged eavesdropping scenario to be economically and
technically unfeasible [51]. As an alternative explanation for suspiciously relevant ads,
he points to the many established and well-documented methods that companies
successfully use to track, prole and micro-target potential customers. Yet another
possible explanation states that the frequently reported phenomenon is merely a pro-
duct of chance, potentially paired with some form of conrmation bias [41]. Finally,
some commentators also suggest that topics of private conversations are sometimes
inspired by unconsciously processed advertisements, which may later cause the per-
ception of being spied upon when the respective ad is encountered again [28].
Many views, theories and arguments have been put forward in attempt to explain
the curious phenomenon, including experimental results and positions from the
research community. However, a consensus has not yet been reached, not even
regarding the fundamental technical feasibility of the alleged eavesdropping attacks.
Therefore, this paper reviews, veries and compares existing arguments from both
sides of the discourse. Apart from providing a structured overview of the matter,
conclusions about the feasibility and detectability of smartphone-based eavesdropping
are drawn based on existing research and our own analysis.
Is My Phone Listening in? On the Feasibility and Detectability 103
In accordance with the reports found on the phenomenon, this paper will focus on
smartphones specically, iOS and Android devices. Since smartphones are the most
widespread consumer electronics device, and since iOS and Android together clearly
dominate the mobile OS market [70], this choice seems justied to us. However, most
of the considerations in this paper are applicable to other types of mobile devices and
other operating systems as well.
The remainder of this paper is structured as follows. In Sect. 2, we describe the
underlying threat model, distinguishing between three possible adversaries. Section 3
examines the possibility of using smartphone microphones for stealthy eavesdropping,
expanding on aspects of security permissions and user notications. Similarly, Sect. 4
considers smartphone motion sensors as a potential eavesdropping channel, taking into
account sampling frequency limits enforced by mobile operating systems. Section 5
then looks into the effectiveness of existing mitigation and detection techniques
developed by Google, Apple, and the global research community. In Sect. 6, the
ecosystem providers themselves are considered as potential adversaries. Section 7
evaluates the technical and economic feasibility of large-scale eavesdropping attacks.
After that, Sect. 8examines ways in which governmental and criminal hackers can
compromise the speech privacy of smartphone users. Finally, Sect. 9provides a dis-
cussion of analysis results, followed by a conclusion in Sect. 10.
2 Threat Model
To target advertisements based on smartphone eavesdropping, an organization A, who
is responsible for selecting the audience for certain online ads (either the advertiser
itself or a contractor entrusted with this task, such as an advertising network
1
), needs to
somehow gain access to sensor data
2
from the corresponding mobile device, or to
information derived from the sensor data.
Initially, speech is recorded through the smartphone by an actor B, which could be
either (1) the operating system provider itself, e.g. Apple or Google, (2) non-system
apps installed on the device, or (3) third-party libraries
3
included in these apps.
Potentially after some processing and ltering, which can happen locally on the device
or on remote servers, actor B shares relevant information extracted from the recording
directly or through intermediaries with organization A (unless A and B are one and
the same actor, which is also possible).
Organization A then uses the received information to identify the smartphone
owner as a suitable target for specic ads and sends a corresponding broadcast request
to an ad publisher (organization A could also publish the ads itself if it has access to ad
distribution channels). Finally, the publisher displays the ads on websites or apps
either on the smartphone through which the speech was recorded or on other devices
1
Advertising networks are companies that match demand and supply of online ad space by connecting
advertisers to ad publishers. They often hold extensive amounts of data on individual internet users to
enable targeted advertising [17].
2
sensor datacan refer to either audio recordings or motion sensor data (see Sects. 3,4).
3
The role and signicance of third-party apps will be further explained in Sect. 3.1.
104 J. L. Kröger and P. Raschke
that can be linked
4
to the smartphone owner, for example through logins, browsing
behavior, or IP address matching. The websites and apps on which the advertisements
appear do not reveal who recorded the smartphone owners speech. Not even orga-
nization A necessarily understands how and by whom the received proling infor-
mation was initially collected. For illustration, Fig. 1presents a simplied overview of
the threat model.
3 Microphone-Based Eavesdropping
Modern smartphones have the capability to tape any sort of ambient sound through
built-in microphones, including private conversations, and to transmit sensitive data,
such as the recording itself or information extracted from recorded speech, to remote
servers over the Internet. Mobile apps installed on a phone could exploit these capa-
bilities for secret eavesdropping. Aspects concerning app permissions and user noti-
cations that could affect the feasibility and visibility of such an attack are examined in
the following two subsections.
3.1 Microphone Access Permission
Before an app can access microphones in Android and iOS devices, permission has to
be granted by the user. However, people tend to accept such requests blindly if they are
interested in an apps functionality [10]. A survey of 308 Android users found that only
17% of respondents paid attention to permissions during app installation, and no more
than 3% of the participants correctly answered the related comprehension questions
[24].
Fig. 1. A schematic and simplied overview of the threat model.
4
For more information on cross-device tracking, refer to [65].
Is My Phone Listening in? On the Feasibility and Detectability 105
Encouraging app development at the expense of user privacy, current permission
systems are much less strict than they were in early smartphones and have been
criticized as coarse grained and incomplete[59]. Also, once a permission is granted,
it is usually not transparent for users when and for which particular purpose data is
being collected and to which servers it is being sent [62].
To include analytics and advertising capabilities, apps commonly make use of
third-party libraries, i.e., code written by other companies. These libraries share mul-
timedia permissions, such as microphone access, with their corresponding host app and
are often granted direct Internet access [39]. Apart from the concern that third-party
libraries are easily over-privileged, it is considered problematic that app developers
often have limited or no understanding of the library code, which can also be changed
dynamically at runtime [59]. Thus, not only users but also app developers themselves
may be unaware of privacy leaks based on the abuse of granted permissions.
A large variety of existing apps has access to smartphone microphones. Examining
over 17.000 popular Android apps, Pan et al. found that 43.8% ask for permission to
record audio [59].
3.2 User Notications and Visibility
Android and iOS apps with microphone permission can not only record audio at any
time while they are active, i.e. running in the foreground, but also while they are in
background mode, under certain conditions [7,31]. Background apps have limited
privileges and are often suspended to conserve the devices limited resources. In cases,
however, where they request the system to stay alive and continue recording while not
in the foreground, there are ways to indicate this to the user.
In iOS, the status bar will automatically turn bright red when recording takes place
in the background, allowing the user to immediately detect potentially unwanted
microphone activity [8].
While the latest release of Android (version 9 Pie) implements similar measures
[31], some older versions produce no visible indication when background apps access
the microphone [10]. In this context, it might be worth noting that Android has been
widely criticized for its slow update cycle, with hundreds of millions of devices run-
ning on massively outdated versions [56]. Also, quite obviously, notications in the
graphical user interface are only visible as long as the devices screen is not turned off.
And nally, some experimenters have already succeeded in circumventing the noti-
cation requirements for smartphone media recordings [69].
4 Motion Sensor-Based Eavesdropping
Adversaries might be able to eavesdrop on conversations through cell phones without
accessing the microphone. Studies have shown that smartphone motion sensors more
specically, accelerometers and gyroscopes can be sensitive enough to pick up sound
vibrations and possibly even reconstruct speech signals [36,54,79].
106 J. L. Kröger and P. Raschke
4.1 Experimental Research Findings
There are opposing views on whether non-acoustic smartphone sensors capture sounds
at normal conversational loudness. While Anand and Saxena did not notice an apparent
effect of live human speech on motion sensors in several test devices [3], other studies
report very small but measurable effects of machine-rendered speech, signicant
enough to reconstruct spoken words or phrases [54,79].
Using only smartphone gyroscopes, researchers from Israels defense technology
group Rafael and Stanford University were able to capture acoustic signals rich enough
to identify a speakers gender, distinguish between different speakers and, to some
extent, track what was being said [54]. In a similar experiment, Zhang et al. demon-
strated the feasibility of inferring spoken words from smartphone accelerometer read-
ings in real-time, even in the presence of ambient noise and user mobility [79].
According to their evaluation, the achieved accuracies were comparable to microphone-
based hotword detection applications such as Samsung S Voice and Google Now.
Both [79] and [54] have notable limitations. First of all, their algorithms were only
able to detect a small set of predened keywords instead of performing full speech
recognition. Also, the speech in both experiments was produced by loudspeakers or
phone speakers, which may result in acoustic properties different from live human
speech. In [54], the playback device and the recording smartphone even shared a
common surface, leading critics to suggest that the observed effect on sensor readings
was not caused by aerial sound waves, but rather by direct surface vibrations [3]. Also,
in contrast to Zhang et al., this approach only achieved low recognition accuracies,
particularly for speaker-independent hotword detection. By their own admission,
however, the authors of [54] are security experts, not speech recognition experts[32].
Therefore, the study should be regarded as an initial exploration rather than a perfect
simulation of state-of-the-art spying techniques. With regard to the effectiveness of
their approach, the researchers pointed out several possible directions for future
improvement.
It might also be noteworthy that patents have already been led for methods to
capture acoustic signals through motion sensors, including a method of detecting a
users voice activity using an accelerometer[21] and a system that uses an
accelerometer in a mobile device to detect hotwords[55].
4.2 Sampling Frequency Limits
In order to limit energy consumption and because typical applications of smartphone
motion sensors do not require highly sampled data, current mobile operating systems
impose a cap on the sampling frequency of motion sensors, such as a maximum of
200 Hz for accelerometer readings in Android [3] and 100 Hz for gyroscopes in iOS
[32]. For comparison, the fundamental frequency of the human speaking voice typi-
cally lies between 85 Hz and 155 Hz for men and 165 Hz and 255 Hz for women [79].
Thus, if at all, non-acoustic smartphone sensors can only capture a limited range of
speech sounds, which presents a challenge to speech reconstruction attacks.
Is My Phone Listening in? On the Feasibility and Detectability 107
With the help of the aliasing effect explained in [54], however, it is possible to
indirectly capture tones above the enforced frequency limits. Furthermore, experiments
show that motion sensor signals from multiple co-located devices can be merged to
obtain a signal with increased sampling frequency, signicantly improving the effec-
tiveness of speech reconstruction attacks [36]. Two or more smartphones that are
located in proximity to each other and whose sensor readings are shared directly or
indirectly with the same actor may therefore pose an increased threat to speech
privacy.
It should also be noted that motion sensors in smartphones are usually capable of
delivering much higher sampling frequencies (often up to 8 kHz) than the upper
bounds prescribed by mobile operating systems [3]. Researchers already expressed
concern that adversaries might be able to override and thereby exceed the software-
based limits through patching applications or kernel drivers in mobile devices [3,54].
4.3 Sensor Access Permissions and Energy Efciency
While certain hardware components, such as camera, microphone and the GPS chip,
are typically protected by permission mechanisms in mobile operating systems, motion
sensors can be directly accessed by third-party apps in iOS and Android without any
prior notication or request to the user [32,45]. Thus, there is usually no way for
smartphone owners to monitor, let alone control when and for what purposes data from
built-in accelerometers and gyroscopes is collected. Even visited websites can often
access smartphone motion sensors [32]. Exploiting accelerometers and gyroscopes to
intrude user privacy is also much more energy-efcient and thus less conspicuous than
recording via microphone [79].
5 Existing Mitigation and Detection Techniques
Many methods are applied by ecosystem providers and security researchers to screen
mobile apps for vulnerabilities and malicious behavior. The following two subsections
examine existing efforts with regard to their potential impact on the feasibility and
detectability of mobile eavesdropping attacks.
5.1 App Inspections Conducted by Ecosystem Providers
Both iOS and Android apply a combination of static, dynamic and manual analysis to
scan new and existing apps on their respective app market for potential security threats
and to ensure that they operate as advertised [78]. Clearly, as the misbehavior of third-
party apps can ultimately damage their own reputation, the platforms have strong
incentives to detect and prevent abuse attempts.
Nevertheless, countless examples of initially undetected malware and privacy leaks
have shown that the security screenings provided by Google and Apple are not always
successful [19]. Google Plays app inspection process has even been described as
fundamentally vulnerable[29]. In a typical cat-and-mouse game, malicious apps
evolve quickly to bypass newly implemented security measures [63], sometimes by
108 J. L. Kröger and P. Raschke
using unbearably simple techniques[29]. In Android devices from uncertied
manufacturers, malware may even be pre-installed before shipment [14]. Signicant
vulnerabilities have also been found in ofcial built-in apps. Apples FaceTime app, for
example, allowed potential attackers to gain unauthorized access to iPhone cameras and
microphones without any requirement of advanced hacking skills [15].
Leaving security loopholes aside, the existing security mechanisms do not guar-
antee privacy protection in terms of data minimization and transparency. Many mobile
apps collect personal data with no apparent relevance to the advertised functionality
[18,62]. Even well-known apps like Uber have not been prevented from collecting
sensitive user data that is not required for the service they offer [46].
There are also many documented cases of mobile apps using their microphone
access in unexpected ways. An example that has received a lot of media attention
recently is the use of so-called ultrasonic beacons, i.e. high-pitched Morse-style
audio signals inaudible to the human ear which are secretly played in stores or
embedded in TV commercials and other broadcast content in order to be able to
unobtrusively track the location, activities and media consumption habits of consumers
[10]. For this to work, the data subject needs to carry a receiving device that records
and scans ambient sound for relevant ultrasonic signals and sends them back to the
tracking network for automated comparison. A constantly growing number of mobile
apps several hundred already, some of them very popular are using their micro-
phone permission for exactly that purpose, often without properly informing the user
about it [10,47]. These apps, some of which are targeted at children and would not
require audio recording for their core functionality, may even detect sounds while the
phone is locked and carried in a pocket [47]. Even in cases where users are aware that
their phone listens in, it is not clear to them what the audio stream is ltered for exactly
and what information is being exltrated. Thus, the example of ultrasonic beacons
shows how apps that have been approved into Apples App Store and Google Play can
exploit their permissions for dubious and potentially unexpected tracking purposes.
Finally, it should not be overlooked that smartphone apps can also be obtained from
various non-ofcial sources, circumventing Apples and Googles permission systems
and auditing processes [62]. In Android, users are free in choosing the source of their
applications [78]. Following a more restrictive policy, iOS only allows users to install
apps downloaded from the ofcial Apple App Store. However, kernel patches can be
used to gain root access and remove software restrictions in iOS (iOS jailbreaking),
which enables users to install apps from uncertied publishers [62].
5.2 App Inspections Conducted by the Research Community
In addition to the checks conducted by Google and Apple, mobile apps are being
reviewed by a broad community of security and privacy researchers. A wide and
constantly expanding range of manual and automated methods is applied for this
purpose.
Pan et al., for instance, scanned 17,260 popular Android apps from different app
markets for potential privacy leaks [59]. Through examining their media permissions,
privacy policies and outgoing network ows, the researchers tried to identify apps that
upload audio recordings to the Internet without explicitly informing the user about it.
Is My Phone Listening in? On the Feasibility and Detectability 109
While unveiling other serious forms of privacy violations, they found no evidence of
such behavior. Based on these ndings, the widely held suspicion of companies
secretly eavesdropping on smartphone users was already portrayed as refuted in news
headlines [34,80].
However, the study comes with numerous limitations: Apart from considering only
a small fraction of the over 2 million available Android apps, the researchers did not
examine media exltration from app background activity, did not consider the use of
privileged APIs, only tested a limited amount of each apps functionalities for a short
amount of time, used a controlled test environment with no real human interactions, did
not consider iOS apps at all, and were not able to detect media that was intentionally
obfuscated, encrypted at the application-layer, or sent over the network in non-standard
encoding formats. Perhaps most importantly, Pan et al. were not able to rule out the
scenario of apps transforming audio recordings into less detectable text transcripts or
audio ngerprints before sending the information out. This would be a very realistic
attack scenario. In fact, various popular apps are known to compress recorded audio in
such a way [10,33]. While all the choices that Pan et al. made regarding their
experimental setup and methodology are completely understandable and were com-
municated transparently, the limitations do limit the signicance of their ndings. All
in all, their approach would only uncover highly unsophisticated eavesdropping
attempts.
Of course, many other researchers have also tried to detect privacy leaks in iOS and
Android apps [62]. Besides analyzing decompiled code, permission requests and
generated network trafc, other factors, such as battery power consumption and device
memory usage, can also be monitored to detect suspicious app behavior [67]. Although
some experts claim to have observed certain mobile apps recording and sending out
audio with no apparent justication [58], the scientic community has not yet produced
any hard evidence for large-scale eavesdropping through smartphone microphones.
Like the above-cited work by Pan et al., however, other existing methods to
identify privacy threats in mobile devices also come with considerable limitations. Due
to its closed-source nature, there is generally a lack of scalable tools for detecting
malicious apps within iOS [19]. While, on the other hand, numerous efcient methods
have been proposed for automatically scanning Android apps, none of these approaches
is totally effective at detecting privacy leaks [59]. As with security checks of the ofcial
app stores (see Sect. 5.1), there is a wide range of possible obfuscation techniques and
covert channels to circumvent detection mechanisms developed by the scientic
community [10,67]. Furthermore, many of the existing approaches do not indicate if
detected data exltration activities are justied with regard to an apps advertised
functionality [62]. Yerukhimovich et al. even suggest that apps classied as safe or
non-malicious are more likely to leak private information than typical malware[78].
Therefore, the fact that no evidence for large-scale mobile eavesdropping has been
found so far should not be interpreted as an all-clear. It could only mean that it is
difcult under current circumstances perhaps even impossible to detect such attacks
effectively.
110 J. L. Kröger and P. Raschke
6 Ecosystem Providers as Potential Adversaries
Not only third-party apps but also mobile operating systems themselves can access
privacy-sensitive smartphone data and transfer it over the Internet. It has been known
for years that both, iOS and Android, do so extensively [5]. Examining the amount of
data sent back to Googles and Apples servers from test devices, a recent study found
that iPhones on average received four requests per hour from their manufacturer
during idle periods, and eighteen requests during periods of heavy use [68]. Leaving
these numbers far behind, Android phones received forty hourly requests from Google
when in idle state and ninety requests during heavy use. Of course, the number of
requests per hour has only limited informational value. Data is often collected much
more frequently, such as on a secondly basis or even constantly, to be later aggregated,
compressed and sent out in data bundles [5].
While the establishment of network connections can be monitored, many aspects of
data collection and processing in smartphones remain opaque. The source code of iOS
is not made publicly available, and while Android is based on code from the Android
Open Source Project, several of Googles proprietary apps and system components are
closed-source as well [2]. Due to the resulting lack of transparency, it cannot be reliably
ruled out that sensitive data is collected and processed without the will or knowledge of
the smartphone owner although, naturally, this would represent a considerable legal
and reputational risk for the corresponding platform provider.
As an intermediary between applications and hardware resources, operating sys-
tems control the access to smartphone sensors, including microphones, accelerometers
and gyroscopes, and can also decide whether or not sensor activity is indicated to the
user on the devices screen. Other than with third-party apps, there is no superior
authority in the system supervising the actions and decisions of iOS and Android.
While external security experts can carry out inspections using similar methods as
outlined in Sect. 5.2, they also face similar limitations. There is no reason to assume
that operating systems refrain from using sophisticated obfuscation techniques to
conceal their data collection practices. Additionally, being in control of the whole
system, iOS and Android can access data on different levels of their respective software
stack, which gives them more options for stealthy data exltration and could possibly
impede detection.
7 Technical and Economic Feasibility
Even where adversaries manage to get around security measures and evade detection, it
remains questionable whether a continuous and large-scale eavesdropping operation for
the purpose of ad targeting would be technically feasible and economically viable.
Based on estimations of CPU, battery, network transfer and data storage requirements,
some commentators already stated their conclusion that such an operation would be far
too expensive [51,76] and may strain even the resources of the NSA[71]. Taking
into account their underlying assumptions, these estimates appear valid. However, there
are several ways in which smartphone-based eavesdropping could be made much more
efcient and scalable, including:
Is My Phone Listening in? On the Feasibility and Detectability 111
Low quality audio recording. To reduce the required data storage, processing
power and energy consumption, adversaries could record audio at low bitrates.
Speech signals do not even have to be intelligible to the human ear to be recognized
and transcribed into text by algorithms [54].
Local pre-processing. Some steps in the processing of recordings (e.g. transcrip-
tion, extraction of audio features, data ltering, keyword matching, compression)
can be performed locally on the device in order to transmit only the most relevant
data to remote servers and thus reduce network trafc and required cloud storage.
Keyword detection instead of full speech recognition. The amounts of processing
power required for automatic speech recognition can be prohibitively high for local
execution on mobile devices. A less CPU-intensive alternative to full speech
recognition is keyword detection, where only a pre-dened vocabulary of spoken
words is recognized. Such systems can even run on devices with much lower
computational power than smartphones, such as 16-bit microcontrollers [25]. It has
been argued that it would still be too taxing for mobile devices to listen out for the
millions or perhaps billionsof targetable keywords that could potentially be
dropped in private conversations [51]. However, instead of listening for specic
product and brand names, audio recordings can simply be scanned for trigger words
that indicate a persons interest, such as love,enjoyed,orgreat, in order to
identify relevant snippets of the recording, which can then be analyzed in more
depth. In fact, this very audio analysis method has already been patented, with the
specic declared purpose of informing targeted advertising and product recom-
mendations[22].
Selective recording. Instead of recording continuously, an adversary could only
record at selected moments using wake words or triggers based on time, location,
user activity, sound level, and other context variables. This could signicantly
reduce the amount of required storage and network trafc[67].
Mobile apps that use all or some of the above techniques can be light enough to run
smoothly on smartphones, as numerous commercial apps and research projects show
[9,10,33,58,67].
But even if it is possible for companies to listen in on private conversations, some
argue that this information might not be of much value to advertisers, since they would
need to know a conversations context and speaker personalities very well in order to
accurately infer personal preferences and purchase intentions from spoken phrases [51].
This argument is reasonable, but can equally be applied to many other proling
methods, including online tracking and location tracking, which are widely used
nonetheless. Of course, where contextual information is sparse, such methods may lead
to wrong conclusions about the respective data subject, possibly resulting in poor and
inefcient ad targeting. However, this would not conict with the above-mentioned
reports of suspected eavesdropping: While the ads were perceived as inspired by topics
raised in private conversations, they did not always reect the purported witnesses
actual needs and wants [6,12].
From an outside perspective, it cannot be precisely determined how protable
certain types of personal data are for advertisers. It is therefore difcult, if not
impossible, to draw up a meaningful cost-benet calculation. However, it can generally
112 J. L. Kröger and P. Raschke
be assumed that private conversations contain a lot of valuable proling information,
especially when speakers express their interest in certain products or services. It is also
worth mentioning that some of the worlds largest companies earn a signicant portion
of their revenue through advertising for Google and Facebook, this portion amounted
to 85% and 98% in 2018, respectively [1,23]. Prots from advertising can be con-
siderably increased through effective targeting, which requires the collection of detailed
personal information [68]. There is no doubt that smartphone sensor data can be very
useful for this purpose. A recently led patent describes, for example, how local
signalsfrom a mobile device, including motion sensor data and audio data from the
microphone, can be analyzed to personalize a users Facebook news feed [50].
8 Unauthorized Access to Smartphones
Although this is most likely no explanation for suspicious ad placement, it should be
noted that there are many ways in which skilled computer experts or hackerscan gain
unauthorized access to mobile devices. The widespread use of smartphones makes
them a particularly attractive hacking target [4].
Not only cyber criminals, but also law enforcement agencies and secret services
invest heavily in their capabilities to exploit software aws and other security vul-
nerabilities in consumer electronics [73]. It has been known for some time that intel-
ligence agencies, such as NSA, GCHQ, and CIA, are equipped with tools to secretly
compromise devices running iOS, Android and other mobile operating systems,
enabling them to move inside a system freely as if they owned it[66,75].
In addition to accessing sensitive data, such as geo-location, passwords, personal
notes, contacts, and text messages, this includes the ability to turn on a phones
microphone without a users consent or awareness [11]. With the help of specialized
tools, smartphone microphones can even be tapped when the device is (or seems)
switched off [73]. Such attacks can also be successful in high-security environments. In
a recent case, for example, more than 100 Israeli servicemen had their phones infected
with spyware that allowed unknown adversaries to control built-in cameras and
microphones [57].
Besides the United States and some European nations, other developed countries,
such as Russia, Israel and China, also have highly sophisticated spying technology at
their disposal [75]. Less developed countries and other actors can buy digital eaves-
dropping tools from a ourishing industry of surveillance contractors at comparatively
low prices [60]. That not only secret services but also law enforcement agencies in the
US can be authorized to convert smartphones into roving bugsto listen in on private
conversations has been conrmed in a 2012 court ruling [16]. Eavesdropping capa-
bilities of criminal organizations should not be underestimated, either. According to a
report by McAfee and the Center for Strategic and International Studies (CSIS), there
are 20 to 30 cybercrime groups with nation-state levelcapacity in countries of the
former Soviet Union alone [52].
Is My Phone Listening in? On the Feasibility and Detectability 113
9 Discussion
So far, despite signicant research efforts, no evidence has been found to conrm the
widespread suspicion that rms are secretly eavesdropping on smartphone users to
inform ads. To the best of our knowledge, however, the opposite has not been proven
either. While some threat scenarios (e.g. the constant transfer of uncompressed audio
recordings into the cloud) can be ruled out based on existing security measures and
considerations regarding an attacks visibility, cost and technical feasibility, there are
still many security vulnerabilities and a fundamental lack of transparency that poten-
tially leave room for more sophisticated attacks to be successful and remain undetected.
In comparison with the researchers cited in this paper, it can be assumed that certain
companies have signicantly more nancial resources, more training data, and more
technical expertise in areas such as signal processing, data compression, covert
channels, and automatic speech recognition. This is besides unresolved contradictions
between cited studies and large remaining research gaps another reason why existing
work should not be seen as nal and conclusive, but rather as an initial exploration of
the issue.
While this paper focuses on smartphones, it should be noted that microphones and
motion sensors are also present in a variety of other Internet-connected devices,
including not only VR headsets, wearable tness trackers and smartwatches, but also
baby monitors, toys, remote controls, cars, household appliances, laptops, and smart
speakers. Some of these devices may have weaker privacy safeguards than smart-
phones. For instance, they may not ask for user permission before turning on the
microphone or may not impose a limit on sensor sampling frequencies. Numerous
devices, including smart TVs [13], smart speakers [27], and connected toys [26], have
already been suspected to spy on private conversations of their users. Certain smart
home devices, such as home security alarms, may even contain a hidden microphone
without disclosing it in the product specications [44]. For these reasons, it is essential
to also thoroughly examine non-smartphone devices when investigating suspicions of
eavesdropping.
It is quite possible, at the same time, that the fears of advertising companies
eavesdropping on private conversations are unfounded. Besides the widespread attri-
bution to chance, one alternative approach to explaining strangely accurate advertise-
ments points to all the established tracking technologies commonly employed by
advertisers that do not depend on any phone sensors or microphones [51].
Drawing from credit card networks, healthcare providers, insurers, employers,
public records, websites, mobile apps, and many other sources, certain multi-national
corporations already hold billions of individual data points on consumerslocation
histories, browsing behaviors, religious and political afliations, occupations, socioe-
conomic backgrounds, health conditions, personality traits, product preferences, and so
on [17,64]. Although their own search engines, social networks, email services, route
planners, instant messengers, and media platforms already give them intimate insight
into the lives of billions of people, advertising giants like Facebook and Google also
intensively track user behavior on foreign websites and apps. Of the 17.260 apps
examined in [59], for example, 48.22% share user data with Facebook in the
114 J. L. Kröger and P. Raschke
background. Through their analytics services and like buttons, Google and Facebook
can track clicks and scrolls of Internet users on a vast number of websites [17].
The deep and potentially unexpected insights that result from such ubiquitous
surveillance can be used for micro-targeted advertising and might thereby create an
illusion of being eavesdropped upon, especially if the data subject is ill-informed about
the pervasiveness and impressive possibilities of data linkage.
Even without being used for audio snooping, smartphones (in their current con-
guration) allow a large variety of actors to track private citizen in a much more
efcient and detailed way than would ever have been possible in even the most
repressive regimes and police states of the 20th century. At the bottom line, whether
sensitive information is extracted from private conversations or collected from other
sources does not make much difference to the possibilities of data exploitation and the
entailing consequences for the data subject. Therefore, whether justied or not, the
suspicions examined in this paper eventually lead to a very fundamental question:
What degree of surveillance should be considered acceptable for commercial purposes
like targeted advertising? Although this paper cannot offer an answer to this political
question, it should not be forgotten that constant surveillance is by no means a tech-
nical necessity and that, by denition, democracies should design and regulate tech-
nology to primarily reect the values of the public, not commercial interests.
Certainly, the fear of eavesdropping smartphones should never be portrayed as
completely unfounded, as various criminal and governmental actors can gain unau-
thorized access to consumer electronics. Although such attacks are unlikely to result in
targeted advertisement, they equally deprive the user of control over his or her privacy
and might lead to other unpredictable harms and consequences. For example, digital
spying tools have been used to inltrate the smartphones of journalists [49] and human
rights activists [60] for repressive purposes.
Finally, it should be recognized that apart from the linguistic contents of speech
microphones and motion sensors may unexpectedly transmit a wealth of other sensitive
information. Through the lens of advanced analytics, a voice recording can reveal a
speakers identity [53], physical and mental health state [20,37], and personality traits
[61], for example. Accelerometer data from mobile devices may implicitly contain
information about a users location [35], daily activities [48], eating, drinking and
smoking habits [72,74], degree of intoxication [30], gender, age, body features and
emotional state [43] and can also be used to re-construct sequences of text entered into
a device, including passwords [42].
10 Conclusion
After online advertisements seemingly adapted to topics raised in private face-to-face
conversations, many people suspect companies to secretly listen in through their
smartphones. This paper reviewed and analyzed existing approaches to explaining the
phenomenon and examined the general feasibility and detectability of mobile eaves-
dropping attacks. While it is possible, on the one hand, that the strangely accurate ads
were just a product of chance or conventional proling methods, the spying fears were
Is My Phone Listening in? On the Feasibility and Detectability 115
not disproved so far, neither by device manufacturers and ecosystem providers nor by
the research community.
In our threat model, we considered non-system mobile apps, third-party libraries,
and ecosystem providers themselves as potential adversaries. Smartphone microphones
and motion sensors were investigated as possible eavesdropping channels. Taking into
account permission requirements, user notications, sensor sampling frequencies,
limited device resources, and existing security checks, we conclude that under the
current levels of data collection transparency in iOS and Android sophisticated
eavesdropping operations could potentially be run by either of the above-mentioned
adversaries without being detected. At this time, no estimate can be made as to the
probability and economic viability of such attacks.
References
1. Alphabet Inc.: Alphabet Announces Fourth Quarter and Fiscal Year 2018 Results (2019).
https://abc.xyz/investor/static/pdf/2018Q4_alphabet_earnings_release.pdf?cache=adc3b38
2. Amadeo, R.: Googles iron grip on Android: Controlling open source by any means
necessary (2018). https://arstechnica.com/gadgets/2018/07/googles-iron-grip-on-android-
controlling-open-source-by-any-means-necessary/
3. Anand, S.A., Saxena, N.: Speechless: analyzing the threat to speech privacy from
smartphone motion sensors. In: 2018 IEEE Symposium on Security and Privacy, San
Francisco, CA, pp. 10001017. IEEE (2018). https://doi.org/10.1109/SP.2018.00004
4. Aneja, L., Babbar, S.: Research trends in malware detection on Android devices. In: Panda,
B., Sharma, S., Roy, N. (eds.) Data Science and Analytics. Communications in Computer
and Information Science, vol. 799, pp. 629642. Springer, Singapore (2018). https://doi.org/
10.1007/978-981-10-8527-7_53
5. Angwin, J., Valentino-DeVries, J.: Apple, Google Collect User Data (2011). https://www.
wsj.com/articles/SB10001424052748703983704576277101723453610
6. Anonymous: YouTube user demonstrating how Facebook listens to conversations to serve ads
(2017). https://www.reddit.com/r/videos/comments/79i4cj/youtube_user_demonstrating_
how_facebook_listens/
7. Apple: Background Execution. https://developer.apple.com/library/archive/documentation/
iPhone/Conceptual/iPhoneOSProgrammingGuide/BackgroundExecution/
BackgroundExecution.html
8. Apple: Record - iPhone User Guide. https://help.apple.com/iphone/11/?lang=en#/iph4d2a
39a3b
9. Arcas, B.A., et al.: Now playing: continuous low-power music recognition. arXiv Comput.
Res. Repos. abs/1711.10958 (2017). http://arxiv.org/abs/1711.10958
10. Arp, D., et al.: Privacy threats through ultrasonic side channels on mobile devices. In: 2017
IEEE European Symposium on Security and Privacy (EuroS&P), Paris, France, pp. 3547.
IEEE (2017). https://doi.org/10.1109/EuroSP.2017.33
11. Ball, J.: Angry Birds and leakyphone apps targeted by NSA and GCHQ for user data
(2014). https://www.theguardian.com/world/2014/jan/27/nsa-gchq-smartphone-app-angry-
birds-personal-data
12. BBC News Services: Is your phone listening in? Your stories (2017). https://www.bbc.com/
news/technology-41802282
116 J. L. Kröger and P. Raschke
13. Beres, D.: How To Stop Your Smart TV From Eavesdropping On You (2015). https://www.
huffpost.com/entry/your-samsung-tv-is-spying-on-you_n_6647762
14. Bocek, V., Chrysaidos, N.: Android devices ship with pre-installed malware (2018). https://
blog.avast.com/android-devices-ship-with-pre-installed-malware
15. Bogost, I.: FaceTime Is Eroding Trust in Tech (2019). https://www.theatlantic.com/
technology/archive/2019/01/apple-facetime-bug-you-cant-escape/581554/
16. Brown, A.J.: United States v. Oliva (United States Court of Appeals, D.C. No. 3:07-cr-
00050-BR-1) (2012)
17. Christl, W.: Corporate Surveillance in Everyday Life. Cracked Labs, Vienna (2017)
18. Christl, W., Spiekermann, S.: Networks of Control: A Report on Corporate Surveillance,
Digital Tracking, Big Data & Privacy. Facultas, Vienna (2016)
19. Cimitile, A., et al.: Machine learning meets iOS malware: identifying malicious applications
on Apple environment. In: Proceedings of the 3rd International Conference on Information
Systems Security and Privacy, Porto, Portugal, pp. 487492. SciTePress (2017). https://doi.
org/10.5220/0006217304870492
20. Cummins, N., et al.: Speech analysis for health: current state-of-the-art and the increasing
impact of deep learning. Methods (2018). https://doi.org/10.1016/j.ymeth.2018.07.007
21. Dusan, S.V., et al.: System and Method of Detecting a Users Voice Activity Using an
Accelerometer (Patent No.: US9438985B2) (2014). https://patents.google.com/patent/
US9438985B2/en
22. Edara, K.K.: Keyword Determinations from Voice Data (Patent No.: US20140337131A1)
(2014). https://patents.google.com/patent/US20140337131A1/en
23. Facebook: Facebook Reports Fourth Quarter and Full Year 2018 Results. https://s21.q4cdn.
com/399680738/les/doc_nancials/2018/Q4/Q4-2018-Earnings-Release.pdf
24. Felt, A.P., et al.: Android permissions: user attention, comprehension, and behavior. In:
Proceedings of the Eighth Symposium on Usable Privacy and Security (SOUPS 2012),
Washington, D.C. ACM Press (2012). https://doi.org/10.1145/2335356.2335360
25. Fourniols, J.-Y., et al.: An overview of basics speech recognition and autonomous approach
for smart home IOT low power devices. J. Signal Inf. Process. 9, 239257. https://doi.org/
10.4236/jsip.2018.94015
26. de Freytas-Tamura, K.: The Bright-Eyed Talking Doll That Just Might Be a Spy (2018).
https://www.nytimes.com/2017/02/17/technology/cayla-talking-doll-hackers.html
27. Fussell, S.: Behind Every Robot Is a Human (2019). https://www.theatlantic.com/
technology/archive/2019/04/amazon-workers-eavesdrop-amazon-echo-clips/587110/
28. Ganjoo, S.: Is Facebook secretly listening your conversations? New report says yes, security
experts say no proof (2018). https://www.indiatoday.in/technology/features/story/is-
facebook-secretly-listening-your-conversations-new-report-says-yes-security-experts-say-
no-proof-1255870-2018-06-09
29. Gao, G., Chow, M.: Android Applications, Can You Trust Google Play on These. Tufts
University (2016)
30. Gharani, P., et al.: An Articial Neural Network for Gait Analysis to Estimate Blood Alcohol
Content Level. arXiv Comput. Res. Repos. abs/1712.01691 (2017). https://arxiv.org/abs/
1712.01691
31. Google: Android 9 Pie. https://www.android.com/versions/pie-9-0/
32. Greenberg, A.: The Gyroscopes in Your Phone Could Let Apps Eavesdrop on Conversations
(2014). https://www.wired.com/2014/08/gyroscope-listening-hack/
33. Grosche, P., et al.: Audio content-based music retrieval. In: Müller, M., et al. (eds.)
Multimodal Music Processing. Dagstuhl Follow-Ups. Dagstuhl Publishing, Wadern (2012)
Is My Phone Listening in? On the Feasibility and Detectability 117
34. Hale, J.L.: Does Your Smartphone Listen To You? A New Study Debunked This Common
Conspiracy (2018). https://www.bustle.com/p/does-your-smartphone-listen-to-you-a-new-
study-debunked-this-common-conspiracy-9682413
35. Han, J., et al.: ACComplice: location inference using accelerometers on smartphones. In:
2012 Fourth International Conference on Communication Systems and Networks
(COMSNETS), pp. 19 (2012). https://doi.org/10.1109/COMSNETS.2012.6151305
36. Han, J., et al.: PitchIn: eavesdropping via intelligible speech reconstruction using non-
acoustic sensor fusion. In: Proceedings of the 16th ACM/IEEE International Conference on
Information Processing in Sensor Networks (IPSN), pp. 181192. ACM Press, Pittsburgh
(2017). https://doi.org/10.1145/3055031.3055088
37. Hashim, N.W., et al.: Evaluation of voice acoustics as predictors of clinical depression
scores. J. Voice 31(2), 256.e1256.e6 (2017). https://doi.org/10.1016/j.jvoice.2016.06.006
38. Hassan, B.: 1 in 5 Aussies convinced their smartphone is spying on them (2018). https://
www.nder.com.au/press-release-july-2018-1-in-5-aussies-convinced-their-smartphone-is-
spying-on-them
39. He, Y., et al.: Dynamic privacy leakage analysis of Android third-party libraries. In: 1st
International Conference on Data Intelligence and Security (ICDIS), pp. 275280 (2018).
https://doi.org/10.1109/ICDIS.2018.00051
40. Khatibloo, F.: Is Facebook Listening (And So What If They Are)? (2017). https://www.
forbes.com/sites/forrester/2017/03/17/is-facebook-listening-and-so-what-if-they-are/
41. Kleinman, Z.: Is your smartphone listening to you? (2016). https://www.bbc.com/news/
technology-35639549
42. Kröger, J.: Unexpected inferences from sensor data: a hidden privacy threat in the internet of
things. In: Strous, L., Cerf, V.G. (eds.) Internet of Things. Information Processing in an
Increasingly Connected World. IFIP Advances in Information and Communication
Technology, vol. 548, pp. 147159. Springer, Cham (2019). https://doi.org/10.1007/978-
3-030-15651-0_13
43. Kröger, J.L., et al.: Privacy implications of accelerometer data: a review of possible
inferences. In: Proceedings of the 3rd International Conference on Cryptography, Security
and Privacy (ICCSP). ACM, New York (2019). https://doi.org/10.1145/3309074.3309076
44. Lee, D.: Google admits error over hidden microphone (2019). https://www.bbc.com/news/
technology-47303077
45. Liu, X., et al.: Discovering and understanding Android sensor usage behaviors with data
ow analysis. World Wide Web 21(1), 105126 (2018). https://doi.org/10.1007/s11280-
017-0446-0
46. Lomas, N.: Uber to end controversial post-trip tracking as part of privacy drive (2017).
http://social.techcrunch.com/2017/08/29/uber-to-end-controversial-post-trip-tracking-as-
part-of-privacy-drive/
47. Maheshwari, S.: That Game on Your Phone May Be Tracking What Youre Watching on
TV (2017). https://www.nytimes.com/2017/12/28/business/media/alphonso-app-tracking.
html
48. Mannini, A., et al.: Activity recognition using a single accelerometer placed at the wrist or
ankle. Med. Sci. Sports Exerc. 45(11), 21932203 (2013). https://doi.org/10.1249/MSS.
0b013e31829736d6
49. Marczak, B., et al.: Hacking Team and the Targeting of Ethiopian Journalists (2014). https://
citizenlab.ca/2014/02/hacking-team-targeting-ethiopian-journalists/
50. Marra, C.J., et al.: Ranking of News Feed in a Mobile Device Based on Local Signals (Pub.
No.: US20170351675A1) (2017). https://patents.google.com/patent/US20170351675A1/en
51. Martínez, A.G.: Facebooks Not Listening Through Your Phone. It Doesnt Have To (2017).
https://www.wired.com/story/facebooks-listening-smartphone-microphone/
118 J. L. Kröger and P. Raschke
52. McAfee: Net Losses: Estimating the Global Cost of Cybercrime. Center for Strategic and
International Studies (CSIS), Washington, D.C. (2014)
53. McLaren, M., et al.: The 2016 speakers in the wild speaker recognition evaluation. In:
Proceedings of the 16th Annual Conference of the International Speech Communication
Association (INTERSPEECH), pp. 823827 (2016). https://doi.org/10.21437/Interspeech.
2016-1137
54. Michalevsky, Y., et al.: Gyrophone: recognizing speech from gyroscope signals. In:
Proceedings of the 23rd USENIX Security Symposium, pp. 10531067 (2014)
55. Mohapatra, P., et al.: Energy-efcient, Accelerometer-based Hotword Detection to Launch a
Voice-control System. (Patent No.: US20170316779A1) (2017). https://patents.google.com/
patent/US20170316779A1/en
56. Morris, I.: Android Is Still Failing Where Apples iOS Is Winning (2018). https://www.
forbes.com/sites/ianmorris/2018/04/13/android-is-still-failing-where-apples-ios-is-winning/
57. Naor, I.: Breaking The Weakest Link Of The Strongest Chain (2017). https://securelist.com/
breaking-the-weakest-link-of-the-strongest-chain/77562/
58. Nichols, S., Morgans, J.: Your Phone Is Listening and its Not Paranoia (2018). https://www.
vice.com/en_uk/article/wjbzzy/your-phone-is-listening-and-its-not-paranoia
59. Pan, E., et al.: Panoptispy: Characterizing Audio and Video Exltration from Android
Applications. Proc. Priv. Enhanc. Technol. 2018(4), 3350 (2018). https://doi.org/10.1515/
popets-2018-0030
60. Perlroth, N.: Governments Turn to Commercial Spyware to Intimidate Dissidents (2017).
https://www.nytimes.com/2016/05/30/technology/governments-turn-to-commercial-
spyware-to-intimidate-dissidents.html
61. Polzehl, T.: Personality in Speech. Springer, Cham (2015). https://doi.org/10.1007/978-3-
319-09516-5
62. Quattrone, A.: Inferring Sensitive Information from Seemingly Innocuous Smartphone Data.
The University of Melbourne (2016)
63. Rahman, M., et al.: Search rank fraud and malware detection in Google Play. IEEE Trans.
Knowl. Data Eng. 29(6), 13291342 (2017). https://doi.org/10.1109/TKDE.2017.2667658
64. Ramirez, E., et al.: Data Brokers. A Call for Transparency and Accountability. Federal Trade
Commission, Washington, D.C. (2014)
65. Ramirez, R., et al.: Cross-Device Tracking: An FTC Staff Report. Federal Trade
Commission, Washington, D.C. (2017)
66. Rosenbach, M., et al.: iSpy: How the NSA Accesses Smartphone Data (2013). http://www.
spiegel.de/international/world/how-the-nsa-spies-on-smartphones-including-the-blackberry-
a-921161.html
67. Schlegel, R., et al.: Soundcomber: a stealthy and context-aware sound trojan for
smartphones. In: Proceedings of the Network and Distributed System Security Symposium
(NDSS) (2011)
68. Schmidt, D.C.: Google Data Collection. Digital Content Next, New York (2018)
69. Sidor, S.: Exploring limits of covert data collection on Android: apps can take photos with
your phone without you knowing (2014). http://www.ez.ai/2014/05/exploring-limits-of-
covert-data.html)
70. Statista: Global mobile OS market share in sales to end users from 1st quarter 2009 to 2nd
quarter 2018. https://www.statista.com/statistics/266136/global-market-share-held-by-
smartphone-operating-systems/
71. Stern, J.: Facebook Really Is Spyingon You, Just Not Through Your PhonesMic(2018).https://
www.wsj.com/articles/facebook-really-is-spying-on-you-just-not-through-your-phones-mic-
1520448644
Is My Phone Listening in? On the Feasibility and Detectability 119
72. Tang, Q., et al.: Automated detection of pufng and smoking with wrist accelerometers. In:
Proceedings of the 8th International Conference on Pervasive Computing Technologies for
Healthcare. pp. 8087 (2014)
73. Taylor, P.: Edward Snowden interview: Smartphones can be taken over(2015). https://
www.bbc.com/news/uk-34444233
74. Thomaz, E., et al.: A practical approach for recognizing eating moments with wrist-mounted
inertial sensing. In: Proceedings of the ACM International Conference on Ubiquitous
Computing, pp. 10291040. ACM Press (2015). https://doi.org/10.1145/2750858.2807545
75. Timberg, C., et al.: WikiLeaks: The CIA is using popular TVs, smartphones and cars to spy on
their owners (2017). https://www.washingtonpost.com/news/the-switch/wp/2017/03/07/why-
the-cia-is-using-your-tvs-smartphones-and-cars-for-spying/?noredirect=on&utm_term=.c16
2373021c3
76. Triggs, R.: No, your smartphone is not always listening to you (2018). https://www.
androidauthority.com/your-phone-is-not-listening-to-you-884028/
77. Tsukayama, H., Romm, T.: Lawmakers press Apple and Google to explain how they track and
listen to users (2018). https://www.washingtonpost.com/technology/2018/07/09/lawmakers-
press-apple-google-explain-how-they-track-listen-users/
78. Yerukhimovich, A., et al.: Can smartphones and privacy coexist? Assessing technologies
and regulations protecting personal data on Android and iOS devices. MIT Lincoln
Laboratory, Lexington, MA (2016). https://doi.org/10.7249/RR1393
79. Zhang, L., et al.: AccelWord: energy efcient hotword detection through accelerometer. In:
Proceedings of the 13th Annual International Conference on Mobile Systems, Applications,
and Services (MobiSys), pp. 301315. ACM Press (2015). https://doi.org/10.1145/2742647.
2742658
80. No, Phones Arent Listening to Your Conversations, but May Be Recording In-App Videos:
Study (2018). https://www.justandroid.net/2018/07/05/no-phones-arent-listening-to-your-
conversations-but-may-be-recording-in-app-videos-study/
Open Access This chapter is distributed under the terms of the Creative Commons Attribution
4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use,
duplication, adaptation, distribution and reproduction in any medium or format, as long as you
give appropriate credit to the original author(s) and the source, a link is provided to the Creative
Commons license and any changes made are indicated.
The images or other third party material in this chapter are included in the works Creative
Commons license, unless indicated otherwise in the credit line; if such material is not included in
the works Creative Commons license and the respective action is not permitted by statutory
regulation, users will need to obtain permission from the license holder to duplicate, adapt or
reproduce the material.
120 J. L. Kröger and P. Raschke
Article
People report receiving ads on their mobile device that are seemingly related to previous offline conversations (i.e., conversation-related advertising). They may think that this is because their electronic devices are eavesdropping (i.e., e-eavesdropping). To gain insights into the scope and characteristics of conversation-related advertising and e-eavesdropping beliefs, we conducted a survey in the United States (n = 300), the Netherlands (n = 293), and Poland (n = 293). These countries were chosen based on their differences in privacy regulations and history with state surveillance. We find that belief in conversation-related advertising is a widespread cross-country phenomenon, which is higher in the United States compared with European countries. In addition, between half and two-thirds of respondents believe e-eavesdropping is a likely explanation for it. We find that social media is a main contributing factor through which people hear about and most often see conversation-related advertising. Moreover, in response to such advertising, respondents experience more negative than positive affect. The results show that this is a prevalent and timely phenomenon that warrants more research. This carries implications for the (social) media industry and regulators, as it highlights the importance of transparency and (insufficient) individual understanding of data collection and processing.
Article
Against the backdrop of calls for greater platform transparency, this exploratory article investigates Meta’s ‘Why Am I Seeing This Ad’ (WAIST) feature, which is positioned as a consumer-level explanation of Meta’s advertising model. Drawing on our own walkthroughs of Facebook and Instagram and data from the Australian Ad Observatory, we find the feature falls short in two ways. First, the explanations do not always align with how the system and its audience-building tools are sold to and used by advertisers. Second, the feature is focused narrowly on single ads and individual users, doing nothing to generate understanding of the patterns and sequences of targeted advertising in relation to other users or over time. We propose both platform practices and independent research strategies that could help to fill this gap between individual explanations, population-level patterns of targeted online advertising and the societal issues associated with it.
Thesis
Full-text available
This thesis explores the intricate relationship between design and technological development, focusing on Ubiquitous Computing and the interplay between interfaces and data. Ubiquitous Computing, emerging in the late 1990s, aimed for pervasive computational systems enabling seamless interaction with technology. The study employs Research-through-Design, particularly examining Artificial Intelligence (AI) as one of the current applications of the principles of Ubiquitous Computing. The initial research reveals a dominant paradigm of efficiency, ease-of-use, and invisibility, where AI’s data-interface relationship perpetuates endless loops of interaction, driven by the concept of “humans-in-the-loop.” The thesis introduces the concept of “Designs for friction,” prioritizing values like slowness, intrusiveness, and presence overlooked in traditional design processes. The first section provides background knowledge, detailing ubiquitous computing and its application in AI. Then, the term “friction” is formalized as a design concept contesting dominant narratives in design processes. The third part applies “friction as a design setting” in educational contexts, creating case studies to study how such concept is appropriated by design students. Finally, the fourth part discusses research results in comparison to theoretical frameworks. By proposing “Designs for friction”, the thesis encourages designers to reconsider overlooked values in technology design, offering an alternative perspective on the relationship between data, interfaces, and society. The research contributes to the broader discourse on the impact of design on the development of data-driven digital products, such as AI.
Article
Numerous mobile devices are equipped with voice assistants to facilitate contactless user-device interaction. However, the widespread availability of voice assistants also raises security and privacy concerns, as they can be maliciously triggered to perform voice eavesdropping. Although diverse attacks have been taken to manipulate voice assistants for eavesdropping, they exhibit deficiencies of limited attack scopes and conspicuous attack behaviors because they target specific voice assistants or require extra voice commands to activate them. To manipulate arbitrary voice assistants for covert eavesdropping attack, we conduct a comprehensive analysis of voice assistant implementation in the Android system and refine a universal workflow. Through meticulous analysis and experimental verification, we uncover an inherent vulnerability that in voice assistants across device types that can be awakened by an artificial faking Intent. Building on this significant discovery, we propose an attack termed VoiceEar. It leverages a malicious event generation file and a first-in-first-out Intent generation algorithm to trigger voice assistants within the normal workflow for eavesdropping, without voice commands. Finally, we deploy the VoiceEar attacks on 25 mainstream mobile devices, and invite 95 volunteers for eavesdropping activity perception testing. The results unequivocally demonstrate the seamless execution of VoiceEar attacks, with neither users nor devices awareness.
Chapter
Full-text available
Smartphones are equipped with a wide variety of sensors, which can pose significant security and privacy risks if not properly protected. To assess the privacy and security risks of smartphone sensors, we first systematically reviewed 55 research papers. Driven by the findings of the systematic review, we carried out a follow-up questionnaire-based survey on 23 human end-users. The results reflect that the participants have a varying level of familiarity with smartphone sensors, and there is a noticeable dearth of awareness about the potential threats and preventive measures associated with these sensors. The findings from this study will inform the development of effective solutions for addressing security and privacy in mobile devices and beyond.
Article
Full-text available
The article proposes a literature review on how design could be a viable way to make users reflect when using design products in the larger context of data production through digital technologies. Design practitioners con-sider quickness, ease of use, and smoothness as hallmarks of good design that produces digital interfaces that do not disclose what is happening behind the surface, creating opaque situations in which users are not necessarily aware of the consequences of their actions. To reframe this approach to design, we explore the concept of “friction” as a lens to analyze existing definitions of this and related concepts in design and as a metaphorical design approa-ch emerging from the literature. A “frictional” perspective could entail slow interaction with technology or the focus on designing effort in using user interfaces that produce data to dispel opaqueness in existing practices. Using Scopus as a proxy to inquire about the defi-ned term, a corpus of relevant publications is analyzed to gather existing design approaches along with occurring instances of the word “friction” and how it has been used previously. To conclude, we introduce the concepts of “diegetic frictions” and “extra-diegetic frictions” as a possible taxonomy of design interventions that embody the initial intention outlined in the article.
Conference Paper
Full-text available
We show that the MEMS gyroscopes found on modern smartphones are sufficiently sensitive to measure acoustic signals in the vicinity of the phone. The resulting signals contain only very low-frequency information (<200Hz). Nevertheless, we show, using signal processing and machine learning, that this information is sufficient to identify speaker information and even parse speech. Since iOS and Android require no special permissions to access the gyro, our results show that apps and active web content that cannot access the microphone can nevertheless eavesdrop on speech in the vicinity of the phone.
Conference Paper
Full-text available
Accelerometers are sensors for measuring acceleration forces. They can be found embedded in many types of mobile devices, including tablet PCs, smartphones, and smartwatches. Some common uses of built-in accelerometers are automatic image stabilization, device orientation detection, and shake detection. In contrast to sensors like microphones and cameras, accelerometers are widely regarded as not privacy-intrusive. This sentiment is reflected in protection policies of current mobile operating systems, where third-party apps can access accelerometer data without requiring security permission. It has been shown in experiments, however, that seemingly innocuous sensors can be used as a side channel to infer highly sensitive information about people in their vicinity. Drawing from existing literature, we found that accelerometer data alone may be sufficient to obtain information about a device holder's location, activities, health condition, body features, gender, age, personality traits, and emotional state. Acceleration signals can even be used to uniquely identify a person based on biometric movement patterns and to reconstruct sequences of text entered into a device, including passwords. In the light of these possible inferences, we suggest that accelerometers should urgently be re-evaluated in terms of their privacy implications, along with corresponding adjustments to sensor protection mechanisms.
Conference Paper
Full-text available
A growing number of sensors, embedded in wearables, smart electric meters and other connected devices, is surrounding us and reaching ever deeper into our private lives. While some sensors are commonly regarded as privacy-sensitive and always require user permission to be activated, others are less protected and less worried about. However, experimental research findings indicate that many seemingly innocuous sensors can be exploited to infer highly sensitive information about people in their vicinity. This paper reviews existing evidence from the literature and discusses potential implications for consumer privacy. Specifically, the analysis reveals that certain insufficiently protected sensors in smart devices allow inferences about users’ locations, activities and real identities, as well as about their keyboard and touchscreen inputs. The presented findings call into question the adequacy of current sensor access policies. It is argued that most data captured by smart consumer devices should be classified as highly sensitive by default. An introductory overview of sensors commonly found in these devices is also provided, along with a proposed classification scheme.
Article
Full-text available
Automatic speech recognition, often incorrectly called voice recognition, is a computer based software technique that analyzes audio signals captured by a microphone and translates them into machine interpreted text. Speech processing is based on techniques that need local CPU or cloud computing with an Internet link. An activation word starts the uplink; "OK google", "Alexa", … and voice analysis is not usually suitable for autonomous limited CPU system (16 bits microcontroller) with low energy. To achieve this realization , this paper presents specific techniques and details an efficiency voice command method compatible with an embedded IOT low-power device.
Article
Full-text available
The high-fidelity sensors and ubiquitous internet connectivity offered by mobile devices have facilitated an explosion in mobile apps that rely on multimedia features. However, these sensors can also be used in ways that may violate user’s expectations and personal privacy. For example, apps have been caught taking pictures without the user’s knowledge and passively listened for inaudible, ultrasonic audio beacons. The developers of mobile device operating systems recognize that sensor data is sensitive, but unfortunately existing permission models only mitigate some of the privacy concerns surrounding multimedia data. In this work, we present the first large-scale empirical study of media permissions and leaks from Android apps, covering 17,260 apps from Google Play, AppChina, Mi.com, and Anzhi. We study the behavior of these apps using a combination of static and dynamic analysis techniques. Our study reveals several alarming privacy risks in the Android app ecosystem, including apps that over-provision their media permissions and apps that share image and video data with other parties in unexpected ways, without user knowledge or consent. We also identify a previously unreported privacy risk that arises from third-party libraries that record and upload screenshots and videos of the screen without informing the user and without requiring any permissions.
Article
Due to the complex and intricate nature associated with their production, the acoustic-prosodic properties of a speech signal are modulated with a range of health related effects. There is an active and growing area of machine learning research in this speech and health domain, focusing on developing paradigms to objectively extract and measure such effects. Concurrently, deep learning is transforming intelligent signal analysis, such that machines are now reaching near human capabilities in a range of recognition and analysis tasks. Herein, we review current state-of-the-art approaches with speechbased health detection, placing a particular focus on the impact of deep learning within this domain. Based on this overview, it is evident while that deep learning based solutions be become more present in the literature, it has not had the same overall dominating effect seen in other related fields. In this regard, we suggest some possible research directions aimed at fully leveraging the advantages that deep learning can offer speech-based health detection.
Chapter
Mobile phones have become the necessity of modern human life to store our valuable information such as passwords, reminders, messages, photos, videos and social contacts. The advent in mobile technology has made human life easier and more efficient. However, at the same time, our excessive dependency on mobile devices has drawn attention of malware authors and cyber criminals leading to large number of cyber-attacks. Amongst all, the major concern of security threat is on Android smartphones. The key reason for it is that it does not restrict users to download applications from unsafe sites. So, it is important to develop robust and efficient Android Malware detection system in order to protect our sensitive data from cyber-attacks on Android platform. In this work, we discuss different types of Android Malwares and provide critical review on their detection approaches that exist in literature. We also highlight promising new directions of research in the domain of Malware detection on Android devices.