Beyond Self-Report: Tools to Compare Estimated and Real-World Smartphone Use

Abstract

Psychologists typically rely on self-report data when quantifying mobile phone usage, despite little evidence of its validity. In this paper we explore the accuracy of using self-reported estimates when compared with actual smartphone use. We also include source code to process and visualise these data. We compared 23 participants' actual smartphone use over a two-week period with self-reported estimates and the Mobile Phone Problem Use Scale. Our results indicate that estimated time spent using a smartphone may be an adequate measure of use, unless a greater resolution of data are required. Estimates concerning the number of times an individual used their phone across a typical day did not correlate with actual smartphone use. Neither estimated duration nor number of uses correlated with the Mobile Phone Problem Use Scale. We conclude that estimated smartphone use should be interpreted with caution in psychological research.

Full text

RESEARCH ARTICLE
Beyond Self-Report: Tools to Compare
Estimated and Real-World Smartphone Use
Sally Andrews
1
*, David A. Ellis
2,3
, Heather Shaw
3
, Lukasz Piwek
4
1Division of Psychology, Nottingham Trent University, Nottingham, United Kingdom, 2Department of
Psychology, Lancaster University, Lancaster, United Kingdom, 3School of Psychology, University of
Lincoln, Lincoln, United Kingdom, 4Faculty of Business and Law, University of the West of England, Bristol,
United Kingdom
*sally.andrews@ntu.ac.uk
Abstract
Psychologists typically rely on self-report data when quantifying mobile phone usage,
despite little evidence of its validity. In this paper we explore the accuracy of using self-
reported estimates when compared with actual smartphone use. We also include source
code to process and visualise these data. We compared 23 participants’actual smartphone
use over a two-week period with self-reported estimates and the Mobile Phone Problem
Use Scale. Our results indicate that estimated time spent using a smartphone may be an
adequate measure of use, unless a greater resolution of data are required. Estimates con-
cerning the number of times an individual used their phone across a typical day did not cor-
relate with actual smartphone use. Neither estimated duration nor number of uses
correlated with the Mobile Phone Problem Use Scale. We conclude that estimated smart-
phone use should be interpreted with caution in psychological research.
Introduction
Around 2 billion people use smartphones across the globe, with over half the population in
developed countries relying on them daily [1]. This ubiquity means that there is the potential
for objective smartphone data to be used to address research questions in the real world [2].
Indeed, there has been a rapid increase in the number of publications examining the relation-
ship between smartphone use, personality, cognition, health, and behaviour e.g. [3–8]. Despite
this, smartphones themselves have yet to become a standard item in the psychologist’s research
toolbox, and little is known about the validity of self-reported estimates of smartphone use.
Miller recently [9] highlighted how important it is for social science researchers to be cur-
rent with new developments in smartphone research methods. Perhaps the biggest barrier to
exploring the objective (actual) use of smartphone data includes developing suitable apps and
the appropriate tools for processing, analysing and visualising big-data sets [10]. Whereas open
source software to create Android apps is freely available for those with no programming expe-
rience [11], there remains no open source software for analysing and visualising the resulting
data.
PLOS ONE | DOI:10.1371/journal.pone.0139004 October 28, 2015 1/9
OPEN ACCESS
Citation: Andrews S, Ellis DA, Shaw H, Piwek L
(2015) Beyond Self-Report: Tools to Compare
Estimated and Real-World Smartphone Use. PLoS
ONE 10(10): e0139004. doi:10.1371/journal.
pone.0139004
Editor: Jakob Pietschnig, Universitat Wien,
AUSTRIA
Received: June 24, 2015
Accepted: September 8, 2015
Published: October 28, 2015
Copyright: © 2015 Andrews et al. This is an open
access article distributed under the terms of the
Creative Commons Attribution License, which permits
unrestricted use, distribution, and reproduction in any
medium, provided the original author and source are
credited.
Data Availability Statement: All relevant data are
within the paper and its Supporting Information files.
Funding: This work was funded by RIF2014-31 -
Research Investment Fund (University of Lincoln)
(http://www.lincoln.ac.uk/home/research/
researchsupport/researchinvestment/). The funders
had no role in study design, data collection and
analysis, decision to publish, or preparation of the
manuscript.
Competing Interests: The authors have declared
that no competing interests exist.

While self-report data can be collected successfully in situations where it is difficult to obtain
objective data, this may not be an appropriate measure when it comes to estimating smart-
phone use. It remains possible that estimates are sufficient for some research questions. but
much of the cognitive literature on time-perception suggests we are poor at estimating such
durations [12]. Any subjective estimate is also likely to ignore rapid, yet pervasive, checking
behaviours [13].
Here we propose that a simple measure—recording when the phone is in use—can provide
a vast array of information about an individual's daily routine. We describe and explore differ-
ent metrics for objective evaluation of smartphone data, and what this can reveal about smart-
phone use. We include source code for processing, visualising and analysing objective
smartphone data, which can be used by those with little to no programming knowledge. As an
applied example, we then explore the claim that people engage in habitual smartphone check-
ing behaviours, by correlating self-report smartphone use estimates with actual smartphone
use and standardised measures of problem mobile phone use [14]. We finally consider other
research questions that could be explored with this methodology.
Method
Participants
Twenty-nine participants were recruited (17 female, mean age = 22.52, range = 18–33). All par-
ticipants owned Android smartphones and consisted of staff and students at the University of
Lincoln. A priori calculations suggest this number to be adequate for finding a moderate corre-
lation between actual and self-reported use, so we stopped collecting after this number was
reached. The study conformed to the recommendations of the Declaration of Helsinki. All par-
ticipants provided written and oral informed consent after being advised of the purpose of the
study, and the type of data being collected. Approval for the project was obtained from the
School of Psychology Research Ethics Committee at the University of Lincoln. All participants
were reimbursed a small fee (£10) for their time. Two participants were excluded as they had
technological problems partway through the study, while four additional participants were
excluded from the analysis for not providing all self-report estimates.
Materials
Smartphone Application: We developed an Android smartphone app using Funf in a Box [11].
Apps collecting data from Android devices are generated by selecting sensors, and specifying
sampling frequency. We selected the screen on/off option, resulting in a small app that records
a timestamp when a use starts and ends. Data is encrypted and uploaded to a server over Wi-Fi
(for more details see [11]). Our app simply recorded a timestamp when the phone became
active, and a second when this interaction ended (typically screen use, although this also
includes processor intensive activities including calls and playing music).
Mobile Phone Problem Use Scale (MPPUS): This questionnaire consists of 27 items, which
have previously demonstrated positive correlations with self-reported mobile phone use [14].
The MPPUS remains a highly cited scale across health and psychological research [15–19], and
has been used as an additional means of measuring mobile phone use more generally [6,20,
21]) (Cronbach's alpha = .89 for standardised items in our sample).
Procedure
On arrival at the lab, a smartphone application was installed on participants' smartphones.
They were then sent a standardised SMS that they were asked to relay back to the experimenter,
What’s Going On (and Off) with Smartphones?
PLOS ONE | DOI:10.1371/journal.pone.0139004 October 28, 2015 2/9

to determine the length of time taken to check a message. Time taken was recorded from the
notification tone until the message had been relayed. Participants were asked to record an esti-
mate each evening of how long they used their phone that day, for the next 14 consecutive
days. We asked participants to only estimate their phone use during periods where their phone
screen was switched on, as the Funf on-off sensor was advertised as measuring screen state.
However, during testing, it was discovered that the on-off sensor actually measured whether
the phone was in an interactive state, which included activities such as phone calls and listening
to music, commonly done with the screen switched off. While we did not analyse the diary
data further, it is possible that the process influenced participants’later estimations of their
phone use. When participants returned to the lab after 14 days, they were asked to estimate
how much they used their phone on average each day (including calls and listening to music).
This measure was used in subsequent analyses of subjective estimates. They were then asked to
estimate how many times they use their phone each day (number of uses), and finally were
asked to complete the MPPUS. The app was then uninstalled from their device.
Data from the app were converted into a comma separate values file using Funf processing
scripts. This file was further processed using source code to calculate descriptive statistics and
barcode visualisations (as shown in Fig 1; see S1 Appendix for source code). The scripts allow
the user to explore different times of day (morning, afternoon, evening, and night), and to
explore different metrics associated with checking behaviours of different durations (N.B. the
source code requires Matlab 2014b or later). These can be calculated separately for each day, or
across the entire duration of a study. We use descriptive statistics for the first 14 days of the
study throughout. Some timestamps showed very long single use durations (i.e. >5 hours).
Another limitation to the application is that when the phone switches off, the app does not
record the screen turning off. When the phone is turned on again, it also does not record the
screen turning on. This results in a seemingly long ‘on’duration, when the phone itself was
actually turned off. It was therefore unclear whether long durations during the day were as a
result of the phone being in use (e.g. listening to music, or watching a film), or whether the
Fig 1. Barcode of smartphone use over two weeks. Black areas indicate times where the phone was in use and Saturdays are indicated with a red dashed
line. Weekday alarm clock times (and snoozing) are clearly evident.
doi:10.1371/journal.pone.0139004.g001
What’s Going On (and Off) with Smartphones?
PLOS ONE | DOI:10.1371/journal.pone.0139004 October 28, 2015 3/9

phone was turned off. As it is impossible to be sure that all long durations were because of this,
we retain these data in all analyses, and use median values when calculating an average values
for each day, as this is a more accurate summary of the average use length. We then use these
values to calculate the mean use length for each participant. We established that occasions
when this occurred overnight were the result of the phone being turned off. The included
source code (S1 Appendix) enables the visualisation of the data for each participant across all
days, or to create an ‘average heatmap’of one day, seven days, or weekdays and weekends (not
shown here).
Results
Objective Data
The mean daily number of uses and the mean length of these durations (including a median
length for all the durations in a day) and a mean daily duration of phone use (total daily dura-
tion) were calculated for each participant. Participants used their phones a mean of 84.68 times
each day (SD = 55.23) and spent 5.05 hours each day using their smartphone (SD = 2.73).
Length of use was, unsurprisingly, highly skewed, with 55% of all uses less than 30 seconds in
duration (see Fig 2).
We classify ‘checks’as uses up to 15 seconds in duration. To explore these behaviours more
closely, we analysed the percentage of phone interactions with durations under 15 seconds.
These showed three distinct periods of increased use; from 1-3s, 5-6s, and 10–11 seconds. Fig 3
shows a histogram of such checks (in 0.5 second bins). In the lab, mean time taken to unlock
the phone and read a short message was 8.42s (SD = 1.53). With added distractions outside the
lab, the 10-11s time bin is likely to reflect the time taken to read a short message, check the
time or other notifications. We explored whether any of these durations could result from the
Fig 2. Percentage of uses categorised by duration. This illustrates the highly skewed nature of smartphone usage.
doi:10.1371/journal.pone.0139004.g002
What’s Going On (and Off) with Smartphones?
PLOS ONE | DOI:10.1371/journal.pone.0139004 October 28, 2015 4/9

display turning itself off, after a period of being idle. However, results indicated that these
default times did not explain any spike in use (default display off times: mean
LOCKED
=
274.88s, SD
LOCKED
= 842.85s; mean
UNLOCKED
= 282.06s, SD
UNLOCKED
= 524.33s).
We also compared phone use at different times of day; night (00:00–06:00), morning
(06:00–12:00), afternoon (12:00–18:00), and evening (18:00–24:00), as shown in Fig 4. In this
comparison we calculate median duration length—i.e. the median amount of time a user
engaged with their phone before turning the display off—for each participant. Finally, we
explored the total duration spent using the phone at each time of day. For the purposes of this
analysis, phone uses that spanned two time windows (e.g. commencing in the morning and
ending in the afternoon) was allocated to the time period in which it originated.
Three one-way repeated measures ANOVAs (Time of Day; morning, afternoon, evening,
night) were calculated separately for total daily duration, use length, and number of uses. Data
from one participant was removed from total daily duration and median use length analyses, as
they had no data from the night time period. There was a significant difference in the number
of phone uses at different times of day (F(3, 78) = 34.62, p<.001, η
ρ
2
= .571). Tukey's LSD
comparisons revealed more individual uses in the afternoon and evening than in the morning
Fig 3. Number of checks in 0.5 second bins across all participants over a 15 second period. Three
spikes of checking duration are visible.
doi:10.1371/journal.pone.0139004.g003
Fig 4. Participants’mean number of phone uses (a), mean total duration (b), and mean duration length
(c) at different times of day. Error bars show 1 SE from the mean.
doi:10.1371/journal.pone.0139004.g004
What’s Going On (and Off) with Smartphones?
PLOS ONE | DOI:10.1371/journal.pone.0139004 October 28, 2015 5/9

and at night (all ps<.001), that there were more uses in the morning than at night (p<.001),
but that there were no differences in the number of uses between afternoon and evening (p=
.083). Fig 4a shows these differences. There were no significant differences in total daily dura-
tion at different times of day (F(3, 78) = .94, p= .414, η
ρ
2
= .036; see Fig 4b, nor in median use
length (F(3, 78) = 2.33, p= .081, η
ρ
2
= .082; see Fig 4c).
Comparison of objective and subjective measures of smartphone use
We conducted paired-samples t-tests and Pearson correlations to compare actual and esti-
mated smartphone use (see Table 1). For number of phone uses, there were far more actual
phone uses (84.68) than were estimated (37.20; t(23) = 3.93, p<.001), and no significant corre-
lation between the two (r(21) = .11, p= .610) indicating that estimated number of phone uses
does not reflect actual number of uses. For total daily duration there was no significant differ-
ence between actual (5.05 hours) and estimated use (4.12 hours; t(22) = 1.78, p= .086) and
there was a moderate positive correlation between the two (p= .02). This suggests that esti-
mated duration of use may have reasonable relative validity.
We finally compared scores on the MPPUS with objective and estimated smartphone use
and checks using Pearson's correlations (see Table 1). None of these analyses revealed any sig-
nificant relationships (ps>.15). Ten participants scored more than 2SD greater than Bianchi
& Phillips’[14] mean, indicating problem use.
Discussion
Estimated levels of smartphone use have previously been related to sleep, interpersonal rela-
tionships, driving safety, and personality [5,7,22,23]. Here we observe that self-reported esti-
mates of phone use relate moderately to actual behavior in such situations. Conversely,
estimated number of checks showed no clear relationship with actual uses; indeed, actual uses
amounted to more than double the estimated number. It is possible that our limited sample
size obscured a larger effect size. Nevertheless, we suggest that estimated use may not be suffi-
cient if a higher resolution of data are required, but that estimates of total use are likely to be
adequate for many research designs. However, for exploring checking behaviours, estimated
number of uses show little reliability for measuring actual uses.
The quantity of short checking behaviours we observed are comparable with those found by
Oulasvirta and colleagues [24], who collected data in 2009. Smartphone use has become much
more prevalent in the intervening six years, and it would be easy to assume that smartphone
use would increase accordingly. However, our data indicate that checking behaviours are no
more prevalent now than they were six years ago. It is interesting to note that people have little
awareness of the frequency with which they check their phone. Oulasvirta and colleagues made
this claim in 2012, however this is the first paper to demonstrate that rapid mobile phone inter-
actions are habitual [25]. While phone interactions under thirty seconds have previously been
classified as 'checking behaviours', our data suggest that habitual goal-and reward-based
actions are likely to be less than 15 seconds in duration when it comes to checking the time or
message notifications.
In our study, the MPPUS did not correlate with any measure of phone use—actual or esti-
mated. The MPPUS is used not only as a measure of problem phone use, but also as an addi-
tional measure of phone use more generally. To determine validity of the MPPUS for this
purpose, we correlated objective phone use with MPPUS scores. This is not to say that the
MPPUS lacks validity, but rather that people use smartphones for a variety of reasons [26], and
that increased use does not necessitate a problem in itself [27]. It may seem reasonable to
assume that those who spend a long time on their phone have problem mobile phone use.
What’s Going On (and Off) with Smartphones?
PLOS ONE | DOI:10.1371/journal.pone.0139004 October 28, 2015 6/9

However, heavy users are not necessarily the same as problem users. While it is easy to conflate
heavy use with problem use, research into smartphone use should identify heavy use and prob-
lem use independently of one another (e.g. [8]).
Examining how much people actually use their smartphone can be useful for a variety of
applications. For example, all except one of our participants used their phone as an alarm
clock, and most reported that they always use their phone last thing before sleeping. These
usage patterns therefore provide a non-invasive indication of sleep length, which has the poten-
tial to augment sleep diary data [28]. Furthermore, while we have considered usage patterns
across the day, a further extension to this analysis would be to consider how these patterns
across different days of the week. This is likely to have additional social and occupational con-
sequences [29].
Trull and Ebner-Priemer [9] and Miller [10] argue that smartphone data have a great deal
to offer as a research tool in psychology, yet comparatively little research utilises objective
smartphone data. Here we show that estimates of smartphone use have a place within current
research, but we caution that its validity is limited and should be complimented by measure-
ments of real behaviour. We also provide the first method to automatically sample and easily
visualise the frequency of smartphone use with a simple background app. We hope that meth-
ods described in this paper will help overcome some barriers to accessing smartphone data for
research in psychology and that it will form a foundation to build upon in the coming years.
Supporting Information
S1 Appendix. Source Code for analysing smartphone use data. Source code, example
screenprobe.csv data file, and README.txt for processing, visualising and analysing smart-
phone use data. csv2data.mconverts ScreenProbe.csv to usable data, while barcode.mallows
visualisations to be generated. descriptives.mgenerates descriptive statistics that can be used for
quantitative analysis. Source code requires Matlab version 2014b or later, but does not require
any specific toolboxes.
(ZIP)
Author Contributions
Conceived and designed the experiments: SA HS DAE. Performed the experiments: HS. Ana-
lyzed the data: SA. Contributed reagents/materials/analysis tools: SA. Wrote the paper: SA
DAE HS LP.
References
1. McIlroy T. Planet of the phones. The Economist. 2015. Available: http://www.economist.com/news/
leaders/21645180-smartphone-ubiquitous-addictive-and-transformative-planet-phones
Table 1. Correlation matrix of MPPUS scores, and actual and estimated smartphone use.
Estimated uses Actual uses Estimated duration Actual duration
Actual uses 0.11
Estimated duration 0.02 -0.03
Actual duration 0.23 0.12 0.47
a
MPPUS 0.03 0.29 0.17 0.30
a
p= .02
doi:10.1371/journal.pone.0139004.t001
What’s Going On (and Off) with Smartphones?
PLOS ONE | DOI:10.1371/journal.pone.0139004 October 28, 2015 7/9

2. MacKerron G, Mourato S. Happiness is greater in natural environments. Global Environmental Change.
2013; 23(5): 992–1000.
3. Clayton R, Leshner G, Almond A. The Extended iSelf: The Impact of iPhone Separation on Cognition,
Emotion, and Physiology. J Comput-Mediat Comm. 2015; 20(2): 119–135.
4. Dolev-Cohen M, Barak A. Adolescents’use of Instant Messaging as a means of emotional relief. Com-
puters in Human Behavior. 2013; 29(1): 58–63.
5. Feldman G, Greeson J, Renna M, Robbins-Monteith K. Mindfulness predicts less texting while driving
among young adults: Examining attention- and emotion-regulation motives as potential mediators. Per-
sonality and Individual Differences. 2011; 51(7): 856–861. PMID: 22031789
6. Walsh S, White K, Young R. Over-connected? A qualitative exploration of the relationship between
Australian youth and their mobile phones. Journal of Adolescence. 2008; 31(1): 77–92. PMID:
17560644
7. Walsh S, White K, Young RMcD. Needing to connect: The effect of self and others on young people's
involvement with their mobile phones. Aus J of Psych. 2010; 62(4): 194–203.
8. White A, Buboltz W, Igou F. Mobile phone use and sleep quality and length in college students. Interna-
tional Journal of Humanities and Social Science. 2011; 1(18): 51–58.
9. Miller G. The Smartphone Psychology Manifesto. Perspectives on Psychological Science. 2012; 7(3):
221–237. doi: 10.1177/1745691612441215 PMID: 26168460
10. Trull T, Ebner-Priemer U. The Role of Ambulatory Assessment in Psychological Science. Current
Directions in Psychological Science. 2014; 23(6): 466–470. PMID: 25530686
11. Aharony N, Pan W, Ip C, Khayal I, Pentland A. Social fMRI: Investigating and shaping social mecha-
nisms in the real world. Pervasive and Mobile Computing. 2011; 7(6): 643–659.
12. Grondin S. Timing and time perception: A review of recent behavioral and neuroscience findings and
theoretical directions. Attention, Perception, & Psychophysics. 2010; 72(3): 561–582.
13. Rachman S. A cognitive theory of compulsive checking. Behaviour Research and Therapy. 2002; 40
(6): 625–639. PMID: 12051482
14. Bianchi A, Phillips J. Psychological Predictors of Problem Mobile Phone Use. CyberPsychology &
Behavior. 2005; 8(1): 39–51.
15. Hadlington LJ. Cognitive failures in daily life: Exploring the link with Internet addiction and problematic
mobile phone use. Computers in Human Behavior. 2015; 51: 75–81.
16. Leung L. Linking psychological attributes to addiction and improper use of the mobile phone among
adolescents in Hong Kong. Journal of Children and Media. 2008; 2(2): 93–113.
17. Lin TTC, Chiang Y, Jiang C. Sociable people beware? Investigating smartphone vs non-smartphone
dependency symptoms among young Singaporeans. Social Behavior and Personality. 2015; 43 (7):
1209–1216.
18. Öz F, ArslantaşD, Buğrul N, Koyuncu T, Ünsal A. Evaluation of problematic use of mobile phones and
quality of sleep among high school students. International Journal of Human Sciences. 2015; 12(1):
226–235.
19. Takao M, Takahashi S, and Kitamura M. Addictive personality and problematic mobile phone use.
CyberPsychology & Behavior 12.5 (2009): 501–507.
20. Kalhori SM, Mohammadi MR, Alavi SS, Jannatifard F, Sepahbodi G, Reisi MB, et al. Validation and
Psychometric Properties of Mobile Phone Problematic Use Scale (MPPUS) in University Students of
Tehran. Iranian Journal of Psychiatry. 2015; 10(1): 25–31. PMID: 26005477
21. Lopez-Fernandez O, Honrubia-Serrans ML, Freixa-Blanxart M. Adapting the Mobile Phone Problem
Use Scale for the Mobile Phone Problem Use Scale for adolescents: The English Version. Personality
and Individual Differences. 2014; 60: S24–S47.
22. Butt S, Phillips J. Personality and self reported mobile phone use. Computers in Human Behavior.
2008; 24(2): 346–360.
23. Ehrenberg A, Juckes S, White K, Walsh S. Personality and Self-Esteem as Predictors of Young Peo-
ple's Technology Use. CyberPsychology & Behavior. 2008; 11(6):7 39–741.
24. Oulasvirta A, Rattenbury T, Ma L, Raita E. Habits make smartphone use more pervasive. Pers Ubiquit
Comput. 2011; 16(1): 105–114.
25. Lally P, van Jaarsveld C, Potts H, Wardle J. How are habits formed: Modelling habit formation in the
real world. Eur J Soc Psychol. 2009; 40(6): 998–1009.
26. Smith A. U.S. Smartphone Use in 2015. Pew Research Center: Internet, Science & Tech. 2015. Avail-
able: http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/
What’s Going On (and Off) with Smartphones?
PLOS ONE | DOI:10.1371/journal.pone.0139004 October 28, 2015 8/9

27. Billieux J. Problematic use of the mobile phone: a literature review and a pathways model. Current Psy-
chiatry Reviews. 2012; 8(4): 299–307.
28. Vallières A, Moran C. Actigraphy in the assessment of insomnia. Journal of Sleep. 2003; 26(7): 902–
906. PMID: 14655927
29. Ellis DA, Wiseman R, Jenkins R. Mental representations of weekdays. PLoS ONE. 2015; 10(8):
e0134555. doi: 10.1371/journal.pone.0134555 PMID: 26288194
What’s Going On (and Off) with Smartphones?
PLOS ONE | DOI:10.1371/journal.pone.0139004 October 28, 2015 9/9