ArticlePDF Available

The Effect of Timing and Frequency of Push Notifications on Usage of a Smartphone-Based Stress Management Intervention: An Exploratory Trial


Abstract and Figures

Push notifications offer a promising strategy for enhancing engagement with smartphone-based health interventions. Intelligent sensor-driven machine learning models may improve the timeliness of notifications by adapting delivery to a user's current context (e.g. location). This exploratory mixed-methods study examined the potential impact of timing and frequency on notification response and usage of Healthy Mind, a smartphone-based stress management intervention. 77 participants were randomised to use one of three versions of Healthy Mind that provided: intelligent notifications; daily notifications within pre-defined time frames; or occasional notifications within pre-defined time frames. Notification response and Healthy Mind usage were automatically recorded. Telephone interviews explored participants' experiences of using Healthy Mind. Participants in the intelligent and daily conditions viewed (d = .47, .44 respectively) and actioned (d = .50, .43 respectively) more notifications compared to the occasional group. Notification group had no meaningful effects on percentage of notifications viewed or usage of Healthy Mind. No meaningful differences were indicated between the intelligent and non-intelligent groups. Our findings suggest that frequent notifications may encourage greater exposure to intervention content without deterring engagement, but adaptive tailoring of notification timing does not always enhance their use. Hypotheses generated from this study require testing in future work. Trial registration number: ISRCTN67177737.
Content may be subject to copyright.
The Effect of Timing and Frequency of Push
Notifications on Usage of a Smartphone-
Based Stress Management Intervention: An
Exploratory Trial
Leanne G. Morrison
*, Charlie Hargood
, Veljko Pejovic
, Adam W. A. Geraghty
Scott Lloyd
, Natalie Goodman
, Danius T. Michaelides
, Anna Weston
Mirco Musolesi
, Mark J. Weal
, Lucy Yardley
1Psychology, Faculty of Social, Human, and Mathematical Sciences, University of Southampton,
Southampton, Hampshire, United Kingdom, 2Primary Care and Population Sciences, Faculty of Medicine,
University of Southampton, Southampton, Hampshire, United Kingdom, 3Department of Creative
Technology, Faculty of Science and Technology, Bournemouth University, Poole, Dorset, United Kingdom,
4Faculty of Computer and Information Science, University of Ljubljana, Ljubljana, Slovenia, 5Redcar &
Cleveland Borough Council, Redcar, Yorkshire, United Kingdom, 6Health and Social Care Institute, School
of Health and Social Care, Teesside University, Middlesbrough, Tees Valley, United Kingdom, 7Fuse,
Centre for Translational Research in Public Health, Newcastle University, Newcastle uponTyne, Tyne and
Wear, United Kingdom, 8Centre for Public Policy and Health, School of Medicine, Pharmacy and Health,
Durham University, Stockton on Tees, United Kingdom, 9Gateshead Council, Gateshead, Tyne and Wear,
United Kingdom, 10 Electronics and Computer Science, University of Southampton, Southampton, Hampshire,
United Kingdom, 11 Department of Geography, University College London, London, United Kingdom
Push notifications offer a promising strategy for enhancing engagement with smartphone-
based health interventions. Intelligent sensor-driven machine learning models may improve
the timeliness of notifications by adapting delivery to a user’s current context (e.g. location).
This exploratory mixed-methods study examined the potential impact of timing and frequency
on notification response and usage of Healthy Mind, a smartphone-based stress manage-
ment intervention. 77 participants were randomised to use one of three versions of Healthy
Mind that provided: intelligent notifications; daily notifications within pre-defined time frames;
or occasional notifications within pre-defined time frames. Notification response and Healthy
Mind usage were automatically recorded. Telephone interviews explored participants’ experi-
ences of using Healthy Mind. Participants in the intelligent and daily conditions viewed (d =
.47, .44 respectively) and actioned (d = .50, .43 respectively) more notifications compared to
the occasional group. Notification group had no meaningful effects on percentage of notifica-
tions viewed or usage of Healthy Mind. No meaningful differences were indicated between
the intelligent and non-intelligent groups. Our findings suggest that frequent notifications may
encourage greater exposure to intervention content without deterring engagement, but adap-
tive tailoring of notification timing does not always enhance their use. Hypotheses generated
from this study require testing in future work.
Trial registration number:ISRCTN67177737
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 1 / 15
Citation: Morrison LG, Hargood C, Pejovic V,
Geraghty AWA, Lloyd S, Goodman N, et al. (2017)
The Effect of Timing and Frequency of Push
Notifications on Usage of a Smartphone-Based
Stress Management Intervention: An Exploratory
Trial. PLoS ONE 12(1): e0169162. doi:10.1371/
Editor: Dongmei Li, University of Rochester,
Received: August 22, 2016
Accepted: December 13, 2016
Published: January 3, 2017
Copyright: ©2017 Morrison et al. This is an open
access article distributed under the terms of the
Creative Commons Attribution License, which
permits unrestricted use, distribution, and
reproduction in any medium, provided the original
author and source are credited.
Data Availability Statement: Participants in our
study provided consent to share anonymised data
with the academic community for the purposes of
research. Therefore, our ethical approval dictates
that we must include the restriction that the data is
available on request to bona fide researchers.
Access to request the data will be available through
the following DOI (
SOTON/381004) and through the University of
Southampton repository via
The potential for digital interventions to effect positive behaviour change has been demonstrated
in a number of health domains [1]. Yet, intervention usage is often below desired levels [2,3].
Intervention prompts (e.g. emails, SMS, push notifications) have shown promise for motivating
initial enrolment to health behaviour change interventions [4] and evoking repeated interven-
tion use [57], particularly when prompts contain feedback, theoretically-informed content or
behaviour change techniques [6,8]. Following Fogg’s behavioural model [9], prompts may pro-
vide the necessary trigger to engage with intervention content whereas theoretically informed
prompt content may provide the necessary motivation to do so.
Smartphones enable on-the-go delivery of intervention content via push notifications that
can be delivered at convenient times for the user or when specific intervention content is
needed [10,11]. Notifications can also prompt access to more intensive support provided by
other platforms [12,13]. However, evidence suggests that users can receive in excess of 50 noti-
fications per day from a variety of apps [14]. Research has also indicated that sending addi-
tional push messages in different formats (e.g. email and SMS) may have adverse effects on
desired behaviour compared to the use of just one message type [15]. To increase the likeli-
hood that users will attend to intervention notifications it is vital to first identify the factors
that enhance or undermine notification response.
Qualitative research suggests that apps may be quickly discarded if notifications are per-
ceived to be irritating or intrusive [16]. Notifications appear to be most acceptable when users
are provided with control over if, when, and how they are received, and when notifications are
delivered at convenient times that do not disrupt daily routine [1619]. Current research does
not yet provide precise indications about when these convenient times might be or the thresh-
old for when notifications become irritating and intrusive. To optimise the potential impact of
notifications from any app it is vital to establish: a) when users are most likely to attend and
respond to notifications; b) how many notifications are optimal for increasing engagement.
SMS messages sent at user designated ‘good’ times (versus other random times) were found
to have little impact on receptivity to and perceived timeliness of messages [20]. Instead, recep-
tivity and timeliness of SMS messages was influenced by perceptions of the notification content
(e.g. interest). It is not clear how well these findings translate to perceptions of smartphone noti-
fications or a health behaviour change context where interest in and motivation to attend to
notification content may differ. Tailoring notification delivery to user-designated ‘good’ times
also places unnecessary burden on the user. Evidence suggests that users are not able to success-
fully anticipate timeframes within which they will be available and receptive to receiving notifi-
cations and that convenient moments are not necessarily consistent day to day [13,20].
Intelligent, sensor-driven machine learning algorithms enable the timing and content of
notifications to fit with and adapt to the users’ current context (e.g. location, physical activity,
social interaction, sleep patterns etc.) or health state (e.g. stress, mood, physiological function-
ing) [21]. To ensure the content of sensor-driven notifications is engaged with, a fundamental
question is whether sensor data can determine when users are able and willing to respond to a
notification [2226]. If so, engagement with the content of sensor-driven notifications may be
enhanced. While models developed to date show promise, early research to support their accu-
racy is necessarily conducted in highly controlled, contrived settings where participants are
often incentivised to provide reports on the timeliness of notifications that are received several
times a day. Anticipation of interruptible moments in some models has also relied on the use
of wearable sensors [e.g. 26]. It is not clear how successful these models will be in a naturalistic
context where users may be less inclined to respond to notifications and where it is less feasible
to harness wearable sensors.
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 2 / 15
Funding: The study was funded by the UK
Engineering and Physical Sciences Research
Council (EP/I032673/1: UBhave: Ubiquitous and
social computing for positive behaviour change led
by Professor Lucy Yardley);
uk/. The funders had no role in study design, data
collection and analysis, decision to publish, or
preparation of the manuscript.
Competing Interests: The authors have declared
that no competing interests exist.
To our knowledge, no study has yet examined the impact of sensor driven notifications
informed solely from phone-based sensors and delivered in a real-world public health context.
This exploratory study compares the impact of intelligent, sensor-driven notifications with
non-intelligent notifications sent within pre-determined timeframes. All notifications were pro-
vided by “Healthy Mind”, an Android app-based stress management intervention disseminated
in a UK-based public health setting. The aims of the study were to investigate the potential
impact of notification timing (intelligent versus non-intelligent notifications) and frequency
(daily versus occasional notifications) using a mixed methods approach. Usage patterns pro-
vided indications about the strength of any association between notification delivery and notifi-
cation response or intervention usage. Qualitative data on participants’ experiences of using the
app generated potential explanations for when and why notifications were (not) responded to.
The Healthy Mind intervention has been described in accordance with the TIDieR checklist
[27]. The qualitative components of the study have been reported in accordance with the
COREQ criteria for interviews and focus groups [28].
Participants were randomised post-baseline to one of three versions of Healthy Mind: intelli-
gent, daily, or occasional. Intelligent notifications were triggered at times when the algorithm
predicted that a user was most likely to notice and respond. Opportune times for each user
were identified by sampling data from three phone-based sensors: location (GPS), movement
(accelerometer), and time of day (clock). The first two notifications were triggered at random,
but within designated time and frequency parameters. The timing and frequency of notifica-
tion triggering was then refined after every notification, that is, the app learned when and in
what contexts notifications were responded to most often. Specifically users could receive up
to 3 notifications per day between 08.00 and 22.00 hours. Users could customise the time
range within which notifications were received. Following this learning period, a model of
interruptibility was then built for each user using a Naïve Bayesian classifier that established a
relationship between specific contexts and likelihood of notification response. Once the model
was trained, the user’s context was sampled every 20 minutes to anticipate the likelihood of
notification response.
The classifier utilised location (GPS), movement (accelerometer) and time variables that
were derived from the raw sensor readings. Initially, users’ sampled GPS co-ordinates were
clustered and averaged within particular time-frames to infer “home” (01.00 to 06.00 hours),
“work” (10.00–16.00 hours), and “other” locations. GPS co-ordinates within a 500m radius of
“home” or “work” co-ordinates were then labelled as “home” or “work”. Co-ordinates outside
of this radius were labelled “other”. Accelerometer X, Y, and Z values were collected for 60s
within each 20 minute sampling window. In line with previous activity recognition research,
the mean intensity of acceleration, the variance of acceleration and the mean crossing rate
were then calculated from the raw accelerometer values to provide a proxy measure of move-
ment [29]. Time variables were hour of day and weekend versus weekday.
The classifier labelled the likelihood of notification response as either yes or no based on the
combined values of the sensed variables. A notification was only triggered if a yes label was
returned (i.e. notification response was deemed likely). The relative weighting of each sensed
variable within the classifier varied between users. That is, the model of interruptibility was
personalised to each individual user. A Naïve Bayesian classifier assumed that variables within
each personalised model were unrelated. This means that the relative weighting of each sensed
variable did not vary with respect to other variables in the model. For example, the importance
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 3 / 15
placed on a user’s motion within the classifier remained the same regardless of whether the
user was designated to be at home, work or other location. The sensing, data processing, and
generation of interruptibility models were handled by independent open-source Android
libraries [24,30].
Daily and occasional notifications were triggered randomly within a time range of 17.00 to
20.00 hours. As with notifications, users were able to customise this time range. If the time
frame specified by the participant did not include 17.00 to 20.00 hours then notifications were
triggered at another random time within the limits specified by the participant. The daily ver-
sion triggered one notification within a 24 hour period. The occasional version triggered one
notification within a 72 hour period. The time frame of 17.00 to 20.00 hours was chosen since
prior research has indicated that intervention or notification engagement typically occurs dur-
ing non-working hours [13,17].
Employers were recruited to the study via local UK public health teams, many of whom were
involved in workplace health activities via the North East Better Health at Work award (described
by [31]). Posters, newsletters and email circulars were used to promote the study to employees,
who downloaded Healthy Mind via the Google Play Store. Standard Google guidelines were fol-
lowed to inform participants about what data was being collected. Data collection took place
between September 2014 and February 2015 and the entire study was approved by the University
of Southampton ethics committee and research governance office (approval number: 12156).
Study procedures were fully automated using LifeGuide and Life Guide Toolbox software (http:// Therefore participants provided informed consent to participate in the
study electronically. After downloading the app, participants were presented with a participant
information screen that provided information about the study. Participants were informed that
they could delete the app at any time. Informed consent was provided by clicking ‘next’ on this
screen and continuing to complete the baseline self-report measures. This consent procedure was
approved by the University of Southampton Ethics Committee and Research Governance Office.
Participants were free to use Healthy Mind as often or as little as they wished.
A link to an online feedback survey was sent via email two weeks after initial app download,
which included an invitation to participate in a semi-structured telephone interview. Partici-
pants were sent a further three email reminders to complete the online feedback survey. All
participants who provided consent to be interviewed were contacted via email and/or tele-
phone by LM to arrange the interview. The online feedback survey did not ask participants to
explain their reasons for declining to give consent to be interviewed. First contact between par-
ticipants and interviewers was an email to arrange a convenient date/time to conduct the tele-
phone interview. Thus, no prior relationship between participants and interviewers was
established. All interviews were conducted by a female researcher with prior training and expe-
rience in conducting qualitative interviews (LM). Interviews lasted between 10 and 32 minutes.
Member checks were employed during the interview (e.g. interviewer restated or summarised
participants’ accounts to check understanding and prompt further elaboration). No member
checks were completed after data analysis.
Healthy Mind is a stand-alone Android smartphone application that offers evidence-based
tools for managing stress and other negative emotions (see Fig 1). Healthy Mind was created
using the Life Guide Toolbox software [32]. The tools provided by Healthy Mind were drawn
from mindfulness-based approaches and Cognitive Behavioural Therapy (CBT) (e.g. breathing
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 4 / 15
and meditation practices, monitoring and planning positive experiences, and self-compassion,
see S1 Table for a full list and description of the Healthy Mind tools). The content for Healthy
Mind was adapted from a pre-existing web-based intervention (Healthy Paths) following a
person-based approach [33]. Healthy Paths was originally designed and written by a multidis-
ciplinary team comprised of psychologists and clinicians in close collaboration with individu-
als who were experiencing stressful life circumstances. Healthy Mind was aimed at managing
stress and was not intended as an intervention for psychological disorders (e.g. depression,
A game-based element was introduced to encourage continued usage of the app long
enough for the intelligent triggering system to train (approximately two notification deliver-
ies). Four starter tools were provided when first downloaded (see S2 Table). To unlock the five
further tools, users were asked to rate the helpfulness of a tool each time it was viewed. Once
unlocked, tools were accessible on demand; no restrictions were placed on when or how often
users were able to access each tool.
All notifications consisted of a short teaser invitation (approximately 40 characters) to use
one of the tools, which if clicked on, led to a further screen that offered more information
about the suggested tool (see Table 1). This information screen was designed to support partic-
ipants to overcome barriers to using the tools or provide interesting new facts about how and
why the tool may be helpful. A range of different messages were developed for each tool in the
app to ensure variability of content. The tool suggested within each notification was tailored to
participants’ prior app usage and tool ratings. Three categories of notification were used: tool
announcements, tool suggestions, and general reminders. Tool announcements invited partic-
ipants to try out a newly unlocked tool. Tool suggestions encouraged re-use of tools that par-
ticipants had previously found helpful. General reminders invited the participant to re-use the
app rather than a specific tool. To minimise perceived repetitiveness, the tool suggested was
varied between two consecutive notifications.
Fig 1. Screen shot of Healthy Mind tool menu screen, tool description screen, and tool rating screen (left to
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 5 / 15
At baseline, participants were asked to provide a valid email and complete a short demographic
questionnaire including age, gender, and educational attainment. Baseline measures were kept
intentionally short in order to mimic how individuals usually engage with apps. Usage of
Healthy Mind was automatically recorded using the Life Guide Toolbox software. Table 2 pro-
vides a detailed description of the variables used to characterise notification response and inter-
vention usage in the presented analysis. Semi-structured telephone interviews explored a)
perceptions of Healthy Mind (e.g. likes, dislikes, reactions to notifications), and b) experiences
of using Healthy Mind (e.g. specific tools used, time spent on the app, contexts of use) (see S2
Table for interview schedule). Field notes were taken during and after each interview to take
note of any technical/usability issues with the Healthy Mind app as well as to capture early
thoughts on potential codes for analysis.
Statistical analysis was performed using IBM SPSS Statistics for Windows 21 [34] on usage
data collected within the first 2 weeks after initial app download. Means and standard devia-
tions were computed for continuous variables and n/% computed for categorical variables.
Table 1. Example notification messages.
Notification type Teaser invitation Information screen
A new tool has been
Congratulations, a new tool has been unlocked for you! It’s been a while since you’ve unlocked a new
tool–that’s why we thought you might like a new one to try. Your new tool is: Body Scan. You can unlock
all the Healthy Mind Tools just by using different tools and telling us what you think of them–each time you
rate a tool we’ll unlock a new one for you. There are 9 tools to unlock.
Tool suggestion Do you have 3 minutes? When our lives are hectic we often forget to take some time to ourselves to relax and slow down. The
great thing about taking a breathing space is that you can do it almost anywhere and all you need is 3
spare minutes! Click ‘next’ to give it a try.
General reminder Take time to look after
yourself today
We know it’s difficult to make time to use the Healthy Mind tools when there’s a lot going on in your life.
But this is exactly the time when you need to look after yourself by doing things that help you to feel
happier and healthier. It’s also why we’ve tried to make the Healthy Mind tools quick and easy to use–so
that you don’t need to feel guilty for taking some time out. Click ‘next’ to give the Healthy Mind tools a try.
Table 2. Variables characterising notification response and intervention usage.
Variable Description
The number of notifications received.
The number of notifications viewed (n) and the percentage of notifications viewed
relative to the number received (%).
The number of notifications (n) and the percentage of notifications (%) that were
followed by the action suggested within the notification.
Response delay The delay (in minutes) between when the notification was sent by the triggering
system and when the notification was viewed by the user.
Logins (n) The number of times participants opened the Healthy Mind app either spontaneously
or via a notification.
Login duration The length of time (in minutes) that participants spent on the app during each
separate login.
Total duration The length of time (in minutes) that participants spent on the app.
Tool completion The number of times participants completed a Healthy Mind tool. Tools were defined
as completed if participants viewed the ‘tool rating’ screen.
Days used The number of days on which participants opened the Healthy Mind tool.
Ceased use The proportion of participants who ceased use of Healthy Mind within 2 weeks after
initial download.
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 6 / 15
Since the sample size in this exploratory study did not provide sufficient power to definitively
test for between group differences, results are interpreted as effect sizes with 95% confidence
intervals [35,36]. Eta-square and Cohen’s dwere computed as indications of effect size for con-
tinuous variables. Cramer’s Vwas computed as an indication of effect size for categorical
All telephone interviews were audio-recorded and transcribed verbatim. Inductive thematic
analysis was used to identify recurring patterns and themes relevant to understanding partici-
pants’ experiences of receiving notifications [37]. Data collection and analysis proceeded itera-
tively. The analysis was conducted by LM through a series of phases. First, transcripts were
read and re-read then hand coded line-by-line using ‘in-vivo’ codes wherever possible. This
preliminary set of codes were then organised into a set of potential themes. Constant compari-
son and deviant case analysis were used to identify data that did not fit within potential theme
structure. Themes were subsequently added, merged and/or refined as appropriate. The final
coding and theme structure was discussed and agreed with AW. A paper trail was maintained
throughout all phases of analysis documenting progression from the raw data to the final
theme structure and reported findings.
The analysis was conducted from a realist perspective, assuming that participants’ reports
were a reflection of their genuine attitudes or experiences. This was an exploratory study and
as such the primary analyst (LM) did not hold any pre-conceptions about what themes may
emerge from the qualitative data. That said, the qualitative and quantitative analyses were con-
ducted in parallel. It is therefore possible that emerging findings from the quantitative analyses
influenced interpretation of the qualitative data and the relative salience of emerging themes.
Sample characteristics
In total, 202 participants downloaded Healthy Mind and 162 were randomised to one of the
three notification groups. 40 participants did not complete the baseline measures and so were
not randomised to one of the three notification groups. An early technical error affected the
first 85 randomised participants. 77 participants therefore provided usable data for the pre-
sented analysis (intelligent: n= 25; daily: n= 19; occasional: n= 33).
Just over half the participants were female (n= 48, 62%) with one participant declining to
answer. Age data was missing or suspected to be false (i.e. default selected) for 8 participants.
The age range of the remaining participants was 18 to 62 years (M= 35.94, SD = 10.54). Around
half (n= 41, 53%) of the participants reported university level education (undergraduate or
postgraduate degree), 6 (8%) reported A-level education, 13 (17%) reported GCSE level educa-
tion, 8 (10%) reported attaining a diploma, vocational or professional qualification, 8 (10%)
reported no formal educational qualifications, and 1 declined to answer.
Notification response and intervention usage
On average, seven notifications were received (M= 7.03, SD = 4.94) and two notifications were
viewed (M= 2.16, SD = 3.28) and subsequently actioned (M= 1.71, SD = 3.18). The average
delay between receiving and viewing a notification was just under 3 hours (M= 163 minutes,
SD = 362 minutes). Participants logged in to Healthy Mind between 1 and 26 times (M= 4.56,
SD = 4.8) and used it on between 1 and 12 days (M= 2.96, SD = 2.55). Participants completed
between 0 and 24 tools (M= 3.92, SD = 5.58). The average duration (M) of each login was 4
minutes (SD = 9 minutes) and average total duration of use 19 minutes (SD = 48 minutes).
Just over half the participants stopped using Healthy Mind within 2 weeks post-download
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 7 / 15
(n= 36, 53%). Table 3 presents descriptive statistics (M,SD) for notification response and app
usage by group.
Table 4 presents Cohen’s dfor pairwise group comparisons where η
.01 (small effect).
Medium effects of group were found for the number of notifications viewed, η
= .14 (95% CI
.00 - .33), and actioned, η
= .15 (95% CI .00 - .35). The intelligent and daily groups appeared to
view more notifications (medium effect), and take action on more notifications (medium effect)
compared with the occasional group. Medium effects of group were also found for the number
of logins, η
= .06 (95% CI .00 - .22), the number of days on which the app was used, η
= .06
(95% CI .00 - .22), and delay in viewing a notification, η
= .05 (95% CI .00 - .18). The intelligent
and daily groups appeared to log into the app more (small effect) and have a shorter response
delay (small-medium effect) than the occasional group. The daily group also appeared to use
the app on a greater number of days than the occasional group (small-medium effect).
A small effect of group was found for duration of each login, η
= .02 (95% CI .00 - .10),
total duration of app usage, η
= .01 (95% CI .00 - .06), and the percentage of notifications
actioned, η
= .01 (95% CI .00 - .08). Duration of app use appeared to be shorter in the daily
group compared with the occasional group (small effects). The intelligent group appeared to
Table 3. Descriptive statistics for notification response and app usage by group, M (SD).
Intelligent Daily Occasional
Notifications received 8.08 (6.17) 10.00 (5.28) 4.52 (1.03)
Notifications actioned (n) 2.60 (4.41) 2.05 (3.41) .85 (1.20)
Notifications viewed (n) 2.92 (4.47) 2.63 (3.69) 1.3 (1.24)
Logins 5.44 (7.03) 4.89 (4.14) 3.7 (2.57)
Days used 3.04 (2.94) 3.63 (3.30) 2.52 (1.52)
Response delay (min) 252 (532) 195 (327) 67 (108)
Login duration (mins) 4 (3) 3 (3) 6 (13)
Total duration (mins) 19 (52) 13 (19) 23 (55)
Notifications actioned (%) 25.17 (28.52) 19.00 (26.37) 18.94 (27.83)
Notifications viewed (%) 29.80 (29.06) 28.05 (32.49) 30.45 (30.42)
Tools completion 4.04 (7.05) 3.47 (3.94) 4.09 (5.28)
Table 4. Effect sizes for group comparisons on notification response and app usage.
Intelligent vs daily Intelligent vs occasional Daily vs occasional
d 95% CI d 95% CI d 95% CI
Notifications received -.33*-.93, .27 .76*** .19, 1.31 1.29*** .58, 1.98
Notifications actioned (n) .14 -.46, .73 .50** -.04, 1.04 .43** -.16, 1.00
Notifications viewed (n) .07 -.53, .67 .47** -.07, 1.00 .44** -.15, 1.01
Logins .09 -.51, .69 .31*-.22, .84 .33*-.25, .90
Days used .19 -.79, .41 .24*-.29, .75 .40** -.18, .97
Response delay (min) .10 -.49, .70 .38** -.15, .91 .39** -.20, .97
Login duration (mins) .29*-.31, .89 -.21*-.72, .32 -.28*-.85, .29
Total duration (mins) .16 -.44, .75 -.07 -.59, .45 -.23*-.79, .34
Notifications actioned (%) .23*-.37, .82 .22*-.30, .74 .00 -.12, .12
*** denotes large effect (.8)
** denotes medium (.5) and small-medium effect (.35) and
*denotes small effect (.2) according to Cohen’s guidelines [38].
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 8 / 15
take action on a greater percentage of notifications compared with both the daily and occa-
sional groups (small effect). A small effect of group was also found for the proportion of partic-
ipants ceasing use of Healthy Mind within 2 weeks after initial download, Cramer’s V= .19. A
higher proportion of participants in the intelligent group appeared to cease use of Healthy
Mind (n= 15, 60%) compared with the daily (n= 8, 42%, Cramer’s V= .18) and occasional
(n= 13, 39%, Cramer’s V= .20) groups. Similar proportions of participants appeared to cease
use of Healthy Mind in the daily and occasional groups (Cramer’s V= .03).
The effect of group on the percentage of notifications viewed, η
= .00 (95% CI .00 - .02),
and tool completion was negligible, η
= .00 (95% CI .00 - .04).
Intervention experiences
Seven participants provided consent to be interviewed; 6 participants were subsequently inter-
viewed (intelligent: n= 2, occasional: n= 4) with 1 providing no response to contacts from the
research team. All 6 participants were female, aged between 21 to 52 years of age (M= 34.17,
SD = 11.44). Most were educated to at least degree level (n= 5, 83%). Two participants were
affected by the early technical error affecting intended delivery of notifications. However, they
were included in the qualitative analysis as their experiences of using Healthy Mind could nev-
ertheless provide useful insights of app engagement. Three themes provide insight into partici-
pants’ experiences of notifications.
Notification awareness. A small number of participants accurately reported on the tim-
ing and frequency of notification delivery. Other participants appeared to be unaware of the
notification delivery schedule or reported inaccurate perceptions. Some participants com-
mented that they were happy with the number and type of notifications received. Others
described experiencing frustration in response to a perceived lack of variety in the notification
“And then in the end it got me a bit annoyed, ‘cause I was like, ‘Oh, I’ve done this already—
come on, you know, if you’re going to send me a reminder, like, it’ll be nice if it was some-
thing different.” (P12)
Notifications appeared to be one of a range of factors that encouraged participants to use
the app. Most participants appeared to perceive the notifications as a reminder to use the
Healthy Mind tools. Participants commented that notifications encouraged them to take time
out or stop and think about their day.
“So it’s been really useful for, ‘cause I’m really busy, have two jobs and children and lots of
other stuff, so sometimes you just forget to take time for yourself so. . ., and getting that
reminder, as well, is really good. That kind of thing, oh yeah, I should have a few minutes
just to sort myself out.” (P08)
Participants differed in the extent to which they reported relying on the notifications; some
participants reported using the app only in response to notifications (but not necessarily after
every notification), others reported spontaneous use of the app. One participant explained that
whilst she did not rely on the notifications to remind her to use the tools, the notifications did
prompt her to consider how helpful the tools had been.
Changing relationships. A few participants discussed how the usefulness of notifications
lessened as they became familiar with the app content and more experienced with using the
Healthy Mind tools. One participant explained that after an initial learning period, the tools
were used as and when needed rather than in response to a notification and often without the
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 9 / 15
need to access Healthy Mind. Participants also described quickly working out and sticking to
their favoured tools.
“When I first started using the app, I was using the app and kind of like responding to the
prompts, and then, as I’ve kind of practiced a bit more, I don’t tend to, like, use the actual
app as much–it’s just more that I’ve kind of learnt the techniques that it’s taught me, and I
use them as I need. So yeah, I’d kind of say like my relationship kind of, like, changed.”
Context and fit. The contexts in which notifications were responded to varied between
each participant. Some participants reported using the app and picking up notifications in the
evening to reflect on the day. Other participants reported using the app as a positive start to
the day, during the working day while travelling, or only in response to stressful experiences.
Another participant described using the app while commuting on public transport, which con-
strained use of some of the tools.
“But you know what, I’ve not, I’ve never tried to do that [Healthy Mind] in the right condi-
tions, I think, I’ve kind of thought ‘oh, I’ve got twenty five minutes on the [train], perhaps
we’ll do it then,’ but I kind of feel self-conscious . . .. So again, what, how I haven’t probably
used it is in the privacy of my own home, sitting down really to kind of go through it,
understand about that sort of, learn that kind of relaxation technique, and, and use it in, use
it in that way.” (P11)
Most of the participants reported picking up notifications at times that they perceived to be
most useful or convenient, not necessarily when the app sent them through. Indeed, a couple
of participants discussed their appreciation of the tone of the notifications, which they per-
ceived to offer suggestions rather than overt demands for immediate action.
“It wasn’t kind of like, ‘oh, you’ve got to do this now’ you know? It wasn’t kind of making
demands on your time, it was just kind of like reminding you that, like, these are things that
help you to kind of fight stress.” (P09)
In terms of timing, no meaningful differences were found between intelligent, sensor driven
and pre-determined, static notification delivery. This counters conclusions drawn from
prior research where sensor-driven models have shown slight advantages over non-sensor-
driven comparators [24]. However, prior research has examined sensor-driven models in
artificial experimental settings where participants were incentivised to respond accurately
to arbitrary, survey-based notifications. The contrasting pattern of results observed in the
current study highlights the need to evaluate emerging sensor-driven intervention models
in a variety of contexts, particularly real-world use. In terms of frequency, more notifica-
tions were viewed and actioned in the intelligent and daily groups compared to the occa-
sional group. The percentage of notifications viewed and actioned appeared equivalent
across groups, as did the number of Healthy Mind tools completed. This suggests that send-
ing frequent, daily notifications may not have adverse effects on response rate, nor does it
seem to deter app usage. Sending frequent, daily notifications also means that users are
likely to see more intervention content.
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 10 / 15
Participants in this study appeared to pay little conscious attention to the frequency and
timing of notifications–instead some were demotivated by the perceived repetitiveness of the
notification content despite attempts to provide variety. The influence of notification content
has been noted previously [20] and highlights the need to adequately pilot content [39] to
ensure that it provides a sufficiently interesting and rewarding experience [40].
Response to notifications and usage of Healthy Mind was low across the three notification
groups. On average, participants opened notifications a few hours after receipt and stopped
using Healthy Mind after a few days. It has been suggested by previous research that perceived
social pressure may drive notification response [25]. Indeed, qualitative data from the current
study indicated that notifications were perceived as suggestions for actions that can be ignored
or deferred to a later time, as needed. More frequent response to notifications may be seen for
interventions that incorporate an explicit social or support-based component. Participants in
the current study also described ceasing use of Healthy Mind once they were familiar with the
tools available. This pattern of usage fits with prior qualitative research highlighting individu-
als’ tendency to use apps fleetingly [16] or to “outgrow” apps [41]. A low-intensity, short-term
pattern of usage is not necessarily problematic for all app-based behaviour change interven-
tions. A few days of quick logins may be sufficient to enable users to learn new tools that can
then be practiced without guidance from an intervention.
The sample size did not offer sufficient power to definitively test for between group differ-
ences. The effect sizes reported in this study should be considered tentative and no conclusions
were drawn from small effects given that all confidence intervals crossed zero. The qualitative
sample was also not sufficient to achieve saturation or to compare experiences across the dif-
ferent notification groups. Explanations for the data in this study are hypothesis generating
only and should be used to stimulate further empirical research. While the accuracy of the
intelligent triggering system has been tested and reported elsewhere [24] it was not explicitly
tested within this study. Previous tests of the intelligent triggering system also examined user
interruptibility independently from intervention content. Different notification types (e.g. tool
announcements vs. tool suggestions) may be associated with varied response rates. Notifica-
tion content could not be examined experimentally in this study since the frequency of each
notification type varied according to app usage patterns. New libraries for content-driven noti-
fications have recently been developed [42]. Further empirical research is needed to examine
the effect of notification content and purpose on user receptivity and response in the context
of health behaviour interventions.
The design of this study did not permit us to examine the effect of notification group on
perceived stress or other health-related outcomes. Further research in a larger sample over a
more extended period is needed to identify whether frequency or timing of notification deliv-
ery is associated with health-related change. Finally, this study examined the impact of notifi-
cation timing and frequency for one specific intervention, with one specific implementation of
intelligent sensor-driven notifications. Additional research is needed to examine whether the
same pattern of results is observed for other interventions that may have varied aims, target
behaviour(s) and populations, content, notification types, and sensor-driven data models.
The results from this study suggest that, in naturalistic settings, tailoring notification delivery
to location, movement, and time of day may not always offer any advantage over a priori
assumptions about convenient moments. Smartphones offer a wide range of contextual data
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 11 / 15
that were not utilised in the current study. It may be that alternative combinations of sensor
data will enhance response rates and intervention usage. The results from this study also sug-
gest that sending frequent, daily notifications may not deter users from engaging with an app-
based intervention and could mean that they are exposed to more of the intervention content.
However, precise thresholds for the frequency at which notifications deter or encourage inter-
vention usage are not yet known. For example, it may be that while daily notifications are
acceptable, several notifications per day may be unacceptable. Similarly, too many intelligent,
sensor-driven notifications may be perceived by users as random. Optimal thresholds may
also vary for different population sub-groups and health behaviours. Larger scale studies are
needed to test the hypotheses generated from this study and to examine the impact of other
combinations of sensor data and different notification delivery schedules.
Current approaches to measuring intervention engagement typically rely on objectively
recorded usage data, which may underestimate engagement with the intervention content. It
may be that initial notification receipt or observed app usage provided reminders to practice rel-
evant tools at a later time. Subsequent practice of the tool will not be reflected in the observed
usage patterns. Identifying variables that indicate optimal receptivity to intervention content is
an ongoing challenge for the development of just-in-time adaptive interventions [43]. Nested
qualitative studies can provide more in-depth insight of participants’ experiences following the
intervention and their potential reasons for continued engagement or disengagement [33].
Adoption of a mixed-methods approach to evaluating digital interventions can support more
informed and appropriate conceptualisations about what constitutes poor versus successful
engagement and the factors that underlie whether and when an individual stops using an inter-
vention. Additional work is needed to identify and evaluate novel methods for assessing engage-
ment with digital interventions that can capture off-line activities and experiences.
This exploratory study suggests that tailoring the delivery of notifications based on users’ cur-
rent location and movement may not always encourage greater response rates or intervention
usage in a naturalistic setting compared to sending notifications at assumed good times. This
study also suggests that sending frequent, daily notifications may enhance exposure to inter-
vention content without deterring continued engagement. Additional research is needed to
test the hypotheses generated from this study and to examine whether other types and combi-
nations of phone-based sensor data can enhance the delivery of notifications and subsequent
behaviour change within different health behaviour change interventions. Mixed methods
approaches that combine quantitative and qualitative data can provide a clearer and more
comprehensive picture of user engagement with health behaviour change interventions.
Supporting Information
S1 Table. Healthy Mind Tools.
S2 Table. Interview Schedule.
We would like to thank all participants for agreeing to share their data, views, and experiences.
We would also like to thank the Workplace Health Improvement Specialists who recruited
employers to the study, who in turn supported recruitment of participants.
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 12 / 15
Author Contributions
Conceptualization: LGM CH VP DTM MM MJW LY.
Data curation: LGM CH VP DTM.
Formal analysis: LGM AW.
Funding acquisition: MM MJW LY.
Investigation: LGM CH VP DTM.
Project administration: LGM CH VP.
Software: LGM CH VP DTM.
Supervision: MM MJW LY.
Validation: LGM CH VP AW.
Visualization: LGM.
Writing – original draft: LGM LY.
Writing – review & editing: LGM CH VP AWAG SL NG DTM AW MM MJW LY.
1. Lustria MLA, Noar SM, Cortese J, Van Stee SK, Glueckauf RL, Lee J. A meta-analysis of web-delivered
tailored health behavior change interventions. J Health Commun. 2013; 18(9):1039–1069. doi: 10.
1080/10810730.2013.768727 PMID: 23750972
2. Kohl LFM, Crutzen R, de Vries NK. Online prevention aimed at lifestyle behaviors: a systematic review
of reviews. J Med Internet Res. 2013; 15(7):e146. doi: 10.2196/jmir.2665 PMID: 23859884
3. Pagoto S, Bennett GG. How behavioral science can advance digital health. Transl Behav Med. 2013; 3
(3):271–6. doi: 10.1007/s13142-013-0234-z PMID: 24073178
4. Bardus M, Blake H, Lloyd SL, Suggs S. Reasons for participating and not participating in a e-health
workplace physical activity intervention. Int J Workplace Health Manag. 2014; 7(4):229–246.
5. Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries NK, Brug J, et al. Which intervention character-
istics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A sys-
tematic review. J Med Internet Res. 2011; 13(1):e2. doi: 10.2196/jmir.1639 PMID: 21212045
6. De Leon E, Fuentes LW, Cohen JE. Characterizing periodic messaging interventions across health
behaviors and media: systematic review. J Med Internet Res. 2014; 16(3):e93. doi: 10.2196/jmir.2837
PMID: 24667840
7. Fry JP, Neff RA. Periodic prompts and reminders in health promotion and health behavior interventions:
Systematic review. J Med Internet Res. 2009; 11(2):e16. doi: 10.2196/jmir.1138 PMID: 19632970
8. Morrison LG, Yardley L, Powell J, Michie S. What design features are used in effective e-health inter-
ventions? A review using techniques from Critical Interpretive Synthesis. Telemedicine and e-Health.
2012; 18(2):137–44. doi: 10.1089/tmj.2011.0062 PMID: 22381060
9. Fogg BJ. A Behavior Model for Persuasive Design. In: Chatterjee S, Dev P, editors. Proceedings of the
4th International Conference on Persuasive Technology; 2009 Apr 26–29; Claremont, CA, USA. New
York, NY: ACM; 2009. p. 40.
10. Ben-Zeev D, Schueller SM, Begale M, Duffecy J, Kane JM, Mohr DC. Strategies for mHealth Research:
Lessons from 3 Mobile Intervention Studies. Adm Policy Ment Health. 2014; 42(2):157–167.
11. Danaher BG, Brendryen H, Seeley JR, Tyler MS, Woolley T. From black box to toolbox: Outlining device
functionality, engagement activities, and the pervasive information architecture of mHealth interven-
tions. Internet Interv. 2015; 2(1) :91–101. doi: 10.1016/j.invent.2015.01.002 PMID: 25750862
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 13 / 15
12. Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: Evi-
dence review and recommendations for future research in mental health. Gen Hosp Psychiatry. 2013;
35(4):332–8. doi: 10.1016/j.genhosppsych.2013.03.008 PMID: 23664503
13. Morrison LG, Hargood C, Lin SX, Dennison L, Joseph J, Hughes S, et al. Understanding usage of a
hybrid website and smartphone app for weight management: a mixed-methods study. J Med Internet
Res. 2014; 16(10):e201. doi: 10.2196/jmir.3579 PMID: 25355131
14. Pielot M, Church K, de Oliveira R. An in-situ study of mobile phone notifications. In: Quigley A, Diamond
S, Irani P, Subramanian S, editors. Proceedings of the 16th International Conference on Human-Com-
puter Interaction with Mobile Devices & Services; 2014 Sep 23–26; Tornoto, ON, Canada. New York,
NY: ACM; 2014. p. 233–242.
15. Suggs S, Blake H, Bardus M, Lloyd S. Effects of text messaging in addition to emails on physical activity
among university and college employees in the UK. J Health Serv Res Policy. 2013; 18(Suppl. 1):56–
16. Dennison L, Morrison L, Conway G, Yardley L. Opportunities and challenges for smartphone applica-
tions in supporting health behavior change: qualitative study. J Med Internet Res. 2013; 15(4):e86. doi:
10.2196/jmir.2583 PMID: 23598614
17. Bentley F, Tollmar K. The power of mobile notifications to increase wellbeing logging behavior. Pro-
ceedings of the SIGCHI Conference on Human Factors in Computing Systems; 2013 Apr 27 –May 02;
Paris, France. New York: ACM; 2013.
18. Fukuoka Y, Lindgren T, Jong S. Qualitative exploration of the acceptability of a mobile phone and
pedometer-based physical activity program in a diverse sample of sedentary women. Public Health
Nurs. 2012; 29(3):232–240. doi: 10.1111/j.1525-1446.2011.00997.x PMID: 22512424
19. Marshall AL. Challenges and opportunities for promoting physical activity in the workplace. J Sci Med
Sport. 2004; 7(1):60–66.
20. Fischer JE, Yee N, Bellotti V, Good N, Benford S, Greenhalgh C. Effects of content and time of delivery
on receptivity to mobile interruptions. In: de Sa
´M, Carric¸o L, Correia N, editors. Proceedings of the 12th
International Conference on Human Computer Interaction with Mobile Devices and Services; 2010 Sep
07–10; Lisbon, Portugal. New York, NY: ACM; 2010. p. 103–112.
21. Riley WT, Serrano KJ, Nilsen W, Atienza AA. Mobile and Wireless Technologies in Health Behavior and
the Potential for Intensively Adaptive Interventions. Curr Opin Psychol. 2015; 5: 67–71. doi: 10.1016/j.
copsyc.2015.03.024 PMID: 26086033
22. Fischer JE, Greenhalgh C, Benford S. Investigating episodes of mobile phone activity as indicators of
opportune moments to deliver notifications. In: Byland M, Juhlin O, Fernaeus Y, editors. Proceedings of
the 13th International Conference on Human Computer Interaction with Mobile Devices and Service;
2011 Aug 30 –Sep 02; Stockholm, Sweden. New York, NY: ACM; 2011. p. 181–190.
23. Fogarty J, Hudson SE, Atkeson CG, Avragami D, Forlizzi J, Kiesler S, et al. Predicting human interrupt-
ibility with sensors. ACM Trans Comput-Hum Interact. 2005; 12(1):119–146.
24. Pejovic V, Musolesi, M. InterruptMe. In: Brush AJ, Friday A, Kientz J, Scott J, Song J, editors. Proceed-
ings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing; 2014
Sep 13–17; Seattle, WA, USA. New York, NY: ACM; 2014. p. 897–908.
25. Pielot M, de Oliveira R, Kwak H, Oliver N. Didn’t you see my message? In: Jones M, Palanque P,
Schmidt A, Grossman T, editors. Proceedings of the SIGCHI Conference on Human Factors in Comput-
ing Systems; 2014 Apr 26 –May 01; Toronto, ON, Canada. New York, NY: ACM; 2014. p. 3319–3328.
26. Sarker H, Sharmin M, Ali AA, Rahman M, Bari R, Hossain SM, et al. Assessing the Availability of Users
to Engage in Just-in-Time Intervention in the Natural Environment. In: Brush AJ, Friday A, Kientz J,
Scott J, Song J, editors. Proceedings of the 2014 ACM International Joint Conference on Pervasive and
Ubiquitous Computing; 2014 Sep 13–17; Seattle, WA, USA. New York, NY: ACM; 2014. p. 909–920.
27. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interven-
tions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;
348:g1687. doi: 10.1136/bmj.g1687 PMID: 24609605
28. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-
item checklist for interviews and focus groups. Int J Qual Health Care. 2007; 19(6):349–357. doi: 10.
1093/intqhc/mzm042 PMID: 17872937
29. Lu H, Yang J, Liu Z, Lane ND, Choudhury T, Campbell AT. The Jigsaw continuous sensing engine for
mobile phone applications. In: Beutal J, Ganesan D, Stankovic J, editors. Proceedings of the 8th ACM
Conference on Embedded Networked Sensor Systems; 2010 Nov 03–05; Zurich, Switzerland. New
York, NY: ACM; 2010. p. 71–84.
30. Lathia N, Rachuri K, Mascolo C, Roussos G. Open source smartphone libraries for computational social
science. In: Mattern F, Santini S, Canny JF, Lanheinrich M, Rekimoto J, editors. Proceedings of the
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 14 / 15
2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication; 2013 Sep 08–12;
Zurich, Switzerland. New York, NY: ACM; 2013. p. 911–920.
31. Braun T, Bambra C, Booth M, Adetayo K, Milne E. Better health at work? An evaluation of the effects
and cost–benefits of a structured workplace health improvement programme in reducing sickness
absence. J Public Health. 2015; 37(1):138–42.
32. Hargood C, Michaelides D, Weal M, Pejovic V, Musolesi M, Morrison L, et al. The UBhave framework:
Developing dynamic mobile applications for digital behavioural interventions. Poster presented at: 11th
International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services;
2014 Dec 02–05; London, UK.
33. Yardley L, Morrison L, Bradbury K., Muller I. The person-based approach to intervention development:
Application to digital health-related behavior change interventions. J Med Internet Res. 2015; 17(1):
e30. doi: 10.2196/jmir.4055 PMID: 25639757
34. IBM. SPSS Statistics for Windows (version 21) [Computer software]. Armonk, NY: IBM Corp.
35. Gardner MJ, Altman DG. Confidence intervals rather than P values: estimation rather than hypothesis
testing. Br Med J. 1986; 292:746–750.
36. Maxwell SE. The persistence of underpowered studies in psychological research: causes, conse-
quences, and remedies. Psychol Methods. 2004; 9 (2):147–163. doi: 10.1037/1082-989X.9.2.147
PMID: 15137886
37. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2008; 3(2):77–101.
38. Cohen J. Statistical Power Analysis for the Behavioural Sciences. New York, NY: Academic Press;
39. Willoughby JF, Furberg R. Underdeveloped or underreported? Coverage of pretesting practices and
recommendations for design of text message–based health behavior change interventions. J Health
Commun 2015; 20(4):472–478. doi: 10.1080/10810730.2014.977468 PMID: 25749250
40. Crutzen R, Ruiter R. Interest in behaviour change interventions: A conceptual model. European Health
Psychologist. 2015; 1(1):6–11.
41. Anderson K, Burford O, Emmerton L. Mobile health apps to facilitate self-care: A qualitative study of
user experiences. PLoS One. 2016; 11(5):e0156164. doi: 10.1371/journal.pone.0156164 PMID:
42. Mehrotra A, Musolesi M, Hendley R, Pejovic V. Designing content-driven intelligent notication mecha-
misms for mobile applications. In: Mase K, Lanheinrich M, Gatica-Perez D, Gellersen H, Choudhury T,
Yatani K, editors. Proceedings of the 2015 ACM InternationalJoint Conference on Pervasive and Ubiq-
uitous Computing; 2015 Sep 07–11; Osaka, Japan. New York, NY: ACM; 2015. p. 813–824.
43. Nahum-shani I., Hekler E. B., & Spruijt-Metz D. Building health behaviour models to guide the develop-
ment of just-in-time adaptive interventions: A pragmatic framework. Health Psychol, 2015; 34:1209–19.
Timing and Frequency of Push Notifications
PLOS ONE | DOI:10.1371/journal.pone.0169162 January 3, 2017 15 / 15
... Another stream of research in the field regards optimizing the delivery of notifications, be it in terms of timing, frequency, or context [27,28]. Such research aims to automatize the detection of opportune situations for engagement and personalise the interaction patterns to best fit individual users. ...
... Customising notifications To our knowledge, no study on applications for health and well-being has explicitly explored the role of customisation in regard to push notifications. While some studies utilise applications with such features [2,28], for example, for setting the timing or frequency of notifications, they are not the primary focus of evaluation and minimal information is provided concerning their contribution to performance metrics or about users' attitudes and preferences. ...
... The option to set the time range of notifications can, for example, help users limit their use only to opportune times (e.g., when they are not at work) while setting the desired number can limit unnecessary disturbances. Such features might also be beneficial concerning habituation, as users might only find notifications practical when they first interact with the application, but not after they have become accustomed to it [28]. Furthermore, if the notifications regard different types of content or senders, the user might only wish to opt-in for some but not others. ...
Full-text available
In mobile technologies for health and well-being (mHealth), push notifications are a widely used tool to implement persuasive strategies. However, little research has been carried out to investigate the impact of notifications in such technologies and users’ attitudes towards them. In this study, we address this gap by exploring the role of notifications and their customisation in the context of mobile applications that promote social and mental well-being. Based on observational data collected from 152 participants, we investigate users’ attitudes and preferences towards notifications. Moreover, we explore how users’ attitudes towards notifications differ along demographic dimensions. Our findings show that customisation of notifications is a desired feature that can be used to improve the persuasive aspect of mHealth applications.Implications derived from our study can provide guidance for researchers and practitioners alike when designing app notifications to create better motivating, meaningful, and persuasive experiences.
... The objective of these systems is to deliver timely and relevant notifications, thereby enhancing users' productivity and overall experience. Extensive research has been conducted to explore various aspects of notification systems, including notification timing [33,34], content personalization [35,36], and user preferences [37]. ...
Full-text available
In this paper, we propose the notification optimization method by providing multiple alternative times as a reminder for a forecasted activity with and without probabilistic considerations for the activity that needs to be completed and needs notification. It is important to consider various factors when sending notifications to people after obtaining the results of the forecasted activity. We should not send notifications only when we have forecasted results because future daily activities are unpredictable. Therefore, it is important to strike a balance between providing useful reminders and avoiding excessive interruptions, especially for low probabilities of forecasted activity. Our study investigates the impact of the low probability of forecasted activity and optimizes the notification time with reinforcement learning. We also show the gaps between forecasted activities that are useful for self-improvement by people for the balance of important tasks, such as tasks completed as planned and additional tasks to be completed. For evaluation, we utilize two datasets: the existing dataset and data we collected in the field with the technology we have developed. In the data collection, we have 23 activities from six participants. To evaluate the effectiveness of these approaches, we assess the percentage of positive responses, user response rate, and response duration as performance criteria. Our proposed method provides a more effective way to optimize notifications. By incorporating the probability level of activity that needs to be done and needs notification into the state, we achieve a better response rate than the baseline, with the advantage of reaching 27.15%, as well as than the other criteria, which are also improved by using probability.
... Given the abundance of data linked to false alerts, a Bayes classification predictive model was employed to sort the data and limit the number of false alerts created by the network. Other similar studies include the creation of a psychological intervention that can forecast a patient's reply (Bostock et al., 2019;Morrison et al., 2017). Algorithms have also been used to manage digital limb, a game-based technology that employs the theory of augmented reality to cure limb pains (Prahm et al., 2016). ...
Purpose The goal of this research is to analyse the obstacles to the implementation of mobile health (mHealth) in India and to gain an understanding of the contextual inter-relationships that exist among those obstacles. Design/methodology/approach A combination of a review of the relevant literature and consultation with subject matter experts is used to identify the barriers to widespread adoption of mHealth. The identified barriers are then analysed using an Interpretive Structural Modelling (ISM) technique to study the interplay existing among them and represent these in a hierarchical manner. Findings The study explores a total of 15 factors that reduce the efficiency of mHealth adoption in India. The findings of the MICMAC investigation show that the economic situation of the government, concerns regarding the safety of intellectual technologies, and privacy issues are the primary obstacles because of the significant driving power they have in mHealth applications. Practical implications Promoters of mHealth practices may be able to make better plans if they understand the social barriers and how they affect each other; this leads to easier adoption of these practices. The findings of this study might be helpful for governments of developing nations to produce standards relating to the deployment of mHealth; this will increase the efficiency with which it is adopted. Originality/Contributions At this time, there is no comprehensive analysis of the factors that influence the adoption of mobile health care with social cognitive theory in developing nations like India. In addition, there is a lack of research in investigating how each of these elements affects the success of mHealth activities and how the others interact with them. Because developed nations learned the value of mHealth practices during the recent pandemic, this study, by investigating the obstacles to the adoption of mHealth and their inter-relationships, makes an important addition to both theory and practice.
Purpose There is clear benefit in designing and sending notifications to users that persuade them to interact with an app and marketer goals. The purpose of this study is to examine how different motivational affordances in notifications affects subsequent app use. Design/methodology/approach The authors designed three studies to address the purpose: (1) an online experiment to test how individuals perceived notifications, which contained social affordances, progression-based affordances, and a combination of social and progression affordances; (2) a survey to gain a deeper understanding of why certain notification characteristics were effective and to unearth factors that jointly affected notification effectiveness; and (3) an in-app field experiment to test if the findings from studies 1 and 2 held up in a “real world” setting. Findings The analysis revealed that progression incentives yielded the greatest increases in user behavior. Neither a social incentive, nor a combination of social and progression affordances was more effective than one progression affordance. This effect was heightened by consumers’ involvement with the focal brand. Research limitations/implications The contribution extends knowledge about the use of motivational affordances to gamify push notifications in high-involvement contexts. This implies that greater attention should be paid to how the: length of push notifications, affordances communicated and degree of consumers’ relationship with a focal brand (i.e. involvement) impact notification effectiveness. These findings set out new avenues to investigate the uses of gamification and services marketing in future research. Practical implications The authors provide marketers with insights into the most effective ways to gamify, structure and time the delivery of notifications. In high-involvement contexts where consumers decide whether to act on a gamified marketing affordance quickly, it pays to use push notifications that feature visible, immediate and tangible rewards. Understanding consumers’ involvement with the brand allows marketers to turn notifications from a potential annoyance into a viable conduit for engagement. Originality/value This research extends knowledge on gamification to the domain of push notifications. In doing so, the authors have demonstrated the communicated affordances and wording of the push notifications organizations send affect user behavior. The authors further expand knowledge of the role of consumer involvement on push notification effectiveness while controlling for app usage patterns.
Objectives: This report aims to evaluate the acceptability, appropriateness, feasibility, and adoption of a healthy lunchbox program (SWAP IT), from the parent perspective. Methods: SWAP IT is an mobile health (m-Health) program aimed to support parents in swapping out discretionary foods for healthier alternatives. Following receipt of the program, parents completed validated scales assessing the Acceptability (AIM), Intervention Appropriateness (IAM), and Feasibility (FIM) via a computer-assisted telephone interview (CATI). Parents were asked about their awareness of strategies in SWAP IT and whether the program supported them to make lunchbox swaps from discretionary to everyday foods. Results: Of the 679 parents who consented, 413 completed the CATI (61% response rate). Parent's mean AIM score (out of a total score of 5) was 4.22 (SD 0.48); FIM score was 4.27 (SD 0.54); and IAM score was 4.24 (SD 0.54). Most parents reported receiving the lunchbox messages (54%), with 45% reporting opening all 10 messages and 64% of parents reporting the program helped swap out discretionary foods. Conclusion: The m-Health lunchbox program, SWAP IT, is highly acceptable, easy to adopt, appropriate, and feasible to parents. Implications to public health: Not only is SWAP IT effective, but favourable implementation factors highlight the potential scalability of the program in improving child nutrition.
The article examines the conceptual foundations of the development of digital medicine and studies the possibilities of using digital technologies for the implementation of medical services. Purpose of the research is to reveal the concepts, identify the stages, directions of digitalization in medicine and research the directions of application of digital medical technologies.Methodology of the research is the analysis of secondary information, theoretical sources, the method of grouping and ranking, marketing research using the method of morphological analysis in the form of expert evaluation.Originality/value of the research: the tools of digital medicine are structured and systematized, the most eff ective ways of their application for solving signifi cant problems in the fi eld of medicine are determined, taking into account the research of the theoretical base and the opinions of experts.Findings of the study: the stages of digitalization are systematized, the features of the use of digital technologies in medicine are researched. The main directions of digitalization of medical services are structured and defi ned. The author's contribution lies in the fact that on the basis of the conducted research in the form of morphological analysis, the purpose and possibilities of applying certain areas of digitalization in medicine are more accurately and correctly formulated.
Full-text available
The study sought to evaluate a custom intervention programme, informed by the Theory of Planned Behaviour, which aimed to help youth athletes reduce their problematic mobile phone use. A hybrid mixed-methods design was employed in this study, involving a one-group pretest–posttest design and a mixed-methods multiple case study design. 23 Singaporean youth athletes (aged 13–19 years) participated in the intervention programme, and were assessed at pre, post, and follow-up timepoints for their objective and subjective measures of mobile phone use. After the intervention programme had concluded, a smaller sample of eight participants were selected to participate in semi-structured interviews to get their views and feedback on the intervention programme. The results showed that while there were no significant differences in the objective measures of mobile phone use (i.e., duration and frequency) between the timepoints, there was a significant reduction in the subjective measure (i.e., SAS-SV) of problematic mobile phone use in both post and follow-up timepoints. This was supported by the multiple case study analysis, which found that participants were able to implement the various strategies that were taught during the intervention programme, and that those strategies were able to help them reduce their level of problematic mobile phone use. Overall, participant feedback for the intervention programme was positive, with participants stating that it was fun, interactive, relatable, and informative. There were still some barriers that prevented participants from reducing their problematic mobile phone use, and suggestions were given on how to improve the intervention programme in future.
Full-text available
Objective: Consumers are living longer, creating more pressure on the health system and increasing their requirement for self-care of chronic conditions. Despite rapidly-increasing numbers of mobile health applications ('apps') for consumers' self-care, there is a paucity of research into consumer engagement with electronic self-monitoring. This paper presents a qualitative exploration of how health consumers use apps for health monitoring, their perceived benefits from use of health apps, and suggestions for improvement of health apps. Materials and methods: 'Health app' was defined as any commercially-available health or fitness app with capacity for self-monitoring. English-speaking consumers aged 18 years and older using any health app for self-monitoring were recruited for interview from the metropolitan area of Perth, Australia. The semi-structured interview guide comprised questions based on the Technology Acceptance Model, Health Information Technology Acceptance Model, and the Mobile Application Rating Scale, and is the only study to do so. These models also facilitated deductive thematic analysis of interview transcripts. Implicit and explicit responses not aligned to these models were analyzed inductively. Results: Twenty-two consumers (15 female, seven male) participated, 13 of whom were aged 26-35 years. Eighteen participants reported on apps used on iPhones. Apps were used to monitor diabetes, asthma, depression, celiac disease, blood pressure, chronic migraine, pain management, menstrual cycle irregularity, and fitness. Most were used approximately weekly for several minutes per session, and prior to meeting initial milestones, with significantly decreased usage thereafter. Deductive and inductive thematic analysis reduced the data to four dominant themes: engagement in use of the app; technical functionality of the app; ease of use and design features; and management of consumers' data. Conclusions: The semi-structured interviews provided insight into usage, benefits and challenges of health monitoring using apps. Understanding the range of consumer experiences and expectations can inform design of health apps to encourage persistence in self-monitoring.
Conference Paper
Full-text available
Notifications on mobile phones alert users about new messages, emails, social network updates, and other events. However, little is understood about the nature and effect of such notifications on the daily lives of mobile users. We report from a one-week, in-situ study involving 15 mobile phones users, where we collected real-world notifications through a smartphone logging application alongside subjective perceptions of those notifications through an online diary. We found that our participants had to deal with 63.5 notifications on average per day, mostly from messengers and email. Whether the phone is in silent mode or not, notifications were typically viewed within minutes. Social pressure in personal communication was amongst the main reasons given. While an increasing number of notifications was associated with an increase in negative emotions, receiving more messages and social network updates also made our participants feel more connected with others. Our findings imply that avoiding interruptions from notifications may be viable for professional communication, while in personal communication, approaches should focus on managing expectations.
Conference Paper
Full-text available
An increasing number of notifications demanding the smart-phone user's attention, often arrive at an inappropriate moment , or carry irrelevant content. In this paper we present a study of mobile user interruptibility with respect to notification content, its sender, and the context in which a notification is received. In a real-world study we collect around 70,000 instances of notifications from 35 users. We group notifications according to the applications that initiated them, and the social relationship between the sender and the receiver. Then, by considering both content and context information, such as the current activity of a user, we discuss the design of classi-fiers for learning the most opportune moment for the delivery of a notification carrying a specific type of information. Our results show that such classifiers lead to a more accurate prediction of users' interruptibility than an alternative approach based on user-defined rules of their own interruptibility.
Full-text available
This paper describes an approach that we have evolved for developing successful digital interventions to help people manage their health or illness. We refer to this as the “person-based” approach to highlight the focus on understanding and accommodating the perspectives of the people who will use the intervention. While all intervention designers seek to elicit and incorporate the views of target users in a variety of ways, the person-based approach offers a distinctive and systematic means of addressing the user experience of intended behavior change techniques in particular and can enhance the use of theory-based and evidence-based approaches to intervention development. There are two key elements to the person-based approach. The first is a developmental process involving qualitative research with a wide range of people from the target user populations, carried out at every stage of intervention development, from planning to feasibility testing and implementation. This process goes beyond assessing acceptability, usability, and satisfaction, allowing the intervention designers to build a deep understanding of the psychosocial context of users and their views of the behavioral elements of the intervention. Insights from this process can be used to anticipate and interpret intervention usage and outcomes, and most importantly to modify the intervention to make it more persuasive, feasible, and relevant to users. The second element of the person-based approach is to identify “guiding principles” that can inspire and inform the intervention development by highlighting the distinctive ways that the intervention will address key context-specific behavioral issues. This paper describes how to implement the person-based approach, illustrating the process with examples of the insights gained from our experience of carrying out over a thousand interviews with users, while developing public health and illness management interventions that have proven effective in trials involving tens of thousands of users. Keywords: person-based approach, Internet, qualitative research, evaluation studies, feasibility studies, health promotion, patient education, professional education, behavior change.
Without a complete published description of interventions, clinicians and patients cannot reliably implement interventions that are shown to be useful, and other researchers cannot replicate or build on research findings. The quality of description of interventions in publications, however, is remarkably poor. To improve the completeness of reporting, and ultimately the replicability, of interventions, an international group of experts and stakeholders developed the Template for Intervention Description and Replication (TIDieR) checklist and guide. The process involved a literature review for relevant checklists and research, a Delphi survey of an international panel of experts to guide item selection, and a face-to-face panel meeting. The resultant 12-item TIDieR checklist (brief name, why, what (materials), what (procedure), who intervened, how, where, when and how much, tailoring, modifications, how well (planned), how well (actually carried out)) is an extension of the CONSORT 2010 statement (item 5) and the SPIRIT 2013 statement (item 11). While the emphasis of the checklist is on trials, the guidance is intended to apply across all evaluative study designs. This paper presents the TIDieR checklist and guide, with a detailed explanation of each item, and examples of good reporting. The TIDieR checklist and guide should improve the reporting of interventions and make it easier for authors to structure the accounts of their interventions, reviewers and editors to assess the descriptions, and readers to use the information.
The mobile phone represents a unique platform for interactive applications that can harness the opportunity of an immediate contact with a user in order to increase the impact of the delivered information. However, this accessibility does not necessarily translate to reachability, as recipients might refuse an initiated contact or disfavor a message that comes in an inappropriate moment. In this paper we seek to answer whether, and how, suitable moments for interruption can be identified and utilized in a mobile system. We gather and analyze a real-world smartphone data trace and show that users' broader context, including their activity, location, time of day, emotions and engagement, determine different aspects of interruptibility. We then design and implement InterruptMe, an interruption management library for Android smartphones. An extensive experiment shows that, compared to a context-unaware approach, interruptions elicited through our library result in increased user satisfaction and shorter response times.
Advances in wireless devices and mobile technology offer many opportunities for delivering just-in-time adaptive interventions (JITAIs)-suites of interventions that adapt over time to an individual's changing status and circumstances with the goal to address the individual's need for support, whenever this need arises. A major challenge confronting behavioral scientists aiming to develop a JITAI concerns the selection and integration of existing empirical, theoretical and practical evidence into a scientific model that can inform the construction of a JITAI and help identify scientific gaps. The purpose of this paper is to establish a pragmatic framework that can be used to organize existing evidence into a useful model for JITAI construction. This framework involves clarifying the conceptual purpose of a JITAI, namely, the provision of just-in-time support via adaptation, as well as describing the components of a JITAI and articulating a list of concrete questions to guide the establishment of a useful model for JITAI construction. The proposed framework includes an organizing scheme for translating the relatively static scientific models underlying many health behavior interventions into a more dynamic model that better incorporates the element of time. This framework will help to guide the next generation of empirical work to support the creation of effective JITAIs. (PsycINFO Database Record
Recent advances in mobile and wireless technologies have made real-time assessments of health behaviors and their influences possible with minimal respondent burden. These tech-enabled real-time assessments provide the basis for intensively adaptive interventions (IAIs). Evidence of such studies that adjust interventions based on real-time inputs is beginning to emerge. Although IAIs are promising, the development of intensively adaptive algorithms generate new research questions, and the intensive longitudinal data produced by IAIs require new methodologies and analytic approaches. Research considerations and future directions for IAIs in health behavior research are provided.
Text messaging interventions for health are becoming increasingly popular, but it is unclear how rigorously such interventions are developed and pretested before being implemented. Pretesting is important to the development of successful health communication interventions. This study reviewed the literature published on text messaging health behavior change interventions and examined pretesting practices. Results showed that pretesting is rarely mentioned in articles, although it is not clear as to why. Six articles mentioned pretesting practices conducted for their mobile intervention, and three articles were written specifically on the pretesting of an intervention. Most pretesting used qualitative methods. Recommendations are provided on pretesting best practices and pretesting reporting to help other researchers in the field of mobile health.