ArticlePDF AvailableLiterature Review

Abstract and Figures

“Engagement” with digital behaviour change interventions (DBCIs) is considered important for their effectiveness. Evaluating engagement is therefore a priority; however, a shared understanding of how to usefully conceptualise engagement is lacking. This review aimed to synthesise literature on engagement to identify key conceptualisations and to develop an integrative conceptual framework involving potential direct and indirect influences on engagement and relationships between engagement and intervention effectiveness. Four electronic databases (Ovid MEDLINE, PsycINFO, ISI Web of Knowledge, ScienceDirect) were searched in November 2015. We identified 117 articles that met the inclusion criteria: studies employing experimental or non-experimental designs with adult participants explicitly or implicitly referring to engagement with DBCIs, digital games or technology. Data were synthesised using principles from critical interpretive synthesis. Engagement with DBCIs is conceptualised in terms of both experiential and behavioural aspects. A conceptual framework is proposed in which engagement with a DBCI is influenced by the DBCI itself (content and delivery), the context (the setting in which the DBCI is used and the population using it) and the behaviour that the DBCI is targeting. The context and “mechanisms of action” may moderate the influence of the DBCI on engagement. Engagement, in turn, moderates the influence of the DBCI on those mechanisms of action. In the research literature, engagement with DBCIs has been conceptualised in terms of both experience and behaviour and sits within a complex system involving the DBCI, the context of use, the mechanisms of action of the DBCI and the target behaviour. Electronic supplementary material The online version of this article (doi:10.1007/s13142-016-0453-1) contains supplementary material, which is available to authorized users.
Content may be subject to copyright.
TBM SYSTEMATIC REVIEWS
Conceptualising engagement with digital behaviour change
interventions: a systematic review using principles
from critical interpretive synthesis
Olga Perski ,
1
Ann Blandford,
2
Robert West,
3
Susan Michie
1
Abstract
BEngagement^with digital behaviour change
interventions (DBCIs) is considered important for their
effectiveness. Evaluating engagement is therefore a pri-
ority; however, a shared understanding of how to usefully
conceptualise engagement is lacking. This review aimed
to synthesise literature on engagement to identify key
conceptualisations and to develop an integrative con-
ceptual framework involving potential direct and indirect
influences on engagement and relationships between
engagement and intervention effectiveness. Four elec-
tronic databases (Ovid MEDLINE, PsycINFO, ISI Web of
Knowledge, ScienceDirect) were searched in November
2015. We identified 117 articles that met the inclusion
criteria: studies employing experimental or non-
experimental designs with adult participants explicitly or
implicitly referring to engagement with DBCIs, digital
games or technology. Data were synthesised using prin-
ciples from critical interpretive synthesis. Engagement
with DBCIs is conceptualised in terms of both experiential
and behavioural aspects. A conceptual framework is pro-
posed in which engagement with a DBCI is influenced by
the DBCI itself (content and delivery), the context (the
setting in which the DBCI is used and the population
using it) and the behaviour that the DBCI is targeting. The
context and Bmechanisms of action^may moderate the
influence of the DBCI on engagement. Engagement, in
turn, moderates the influence of the DBCI on those
mechanisms of action. In the research literature, engage-
ment with DBCIs has been conceptualised in terms of
both experience and behaviour and sits within a complex
system involving the DBCI, the context of use, the mech-
anisms of action of the DBCI and the target behaviour.
Keywords
Engagement, Digital, Behaviour change interventions,
eHealth, mHealth, Conceptual framework, Systematic
review
INTRODUCTION
A substantial number of Internet-connected adults use
some forms of digital technology to monitor or mod-
ify their health: estimates vary between 20 and 80%
[13]. Digital behaviour change interventions
(DBCIs), defined as Ba product or service that uses
computer technology to promote behaviour change^
[4], can, for example, be delivered through computer
programs, websites, mobile phones, smartphone
applications (apps) or wearable devices. Evidence sug-
gests that DBCIs can help people change a range of
different health behaviours, including smoking [5,6],
alcohol consumption [7], weight management [8],
physical activity [9] and self-management of chronic
conditions [10]. Some form of Bengagement^with
DBCIs is assumed to be important for their effective-
ness [11]. A positive association between engagement
and, for example, smoking cessation, weight loss and
increased fruit and vegetable intake has been ob-
served [1214]. To date, we have not achieved a
shared understanding of how to usefully conceptualise
and operationalise engagement with DBCIs. This sys-
tematic review, which follows the Cochrane Collabora-
tions Handbook of Systematic Reviews of Interventions [15],
examines how engagement has been construed and
measured in the behavioural science, computer sci-
ence and human-computer interaction (HCI)
1
Department of Clinical, Educational
and Health Psychology,
University College London, 1-19
Torrington Place, London, WC1E 6BT,
UK
2
UCL Interaction Centre,
University College London, 66-72
Gower Street, London, WC1E 6EA,
UK
3
Cancer Research UK, Health
Behaviour Research Centre,
Department of Epidemiology and
Public Health,
University College London, 1-19
Torrington Place, London, WC1E 6BT,
UK
Correspondence to: O Pe rski
olga.perski.14@ucl.ac.uk
doi: 10.1007/s13142-016-0453-1
Implications
Practice: The use of a shared conceptual frame-
work for engagement with digital behaviour
change interventions (DBCIs) should promote
more rapid advances in developing methods to
improve it.
Policy: A shared conceptualisation of engagement
with DBCIs can be used to help policymakers and
commissioners to set standards against which to
evaluate DBCIs.
Research: The proposed conceptual framework
can be used to generate testable hypotheses about
how to improve engagement.
Electronic supplementary material
The online version of this article (doi:10.1007/
s13142-016-0453-1) contains supplementary mate-
rial, which is available to authorized users.
TBM page 1 of 14
literatures and uses this to propose an integrative def-
inition and conceptual framework of engagement with
DBCIs that can be used to generate predictions and
explanations of empirical observations.
The design of DBCIs requires knowledge of inter-
vention content, delivery, interface design and com-
puter programming, which have traditionally been
informed by separate scientific disciplines, such as
behavioural science, computer science and HCI. Sci-
entific disciplines are characterised by accumulating a
body of specialist knowledge and developing a specific
terminology concerned with the particular object of
research [16]. Due to the multifaceted structure of
DBCIs, an interdisciplinary approach, where knowl-
edge from multiple disciplines is harnessed to develop
a shared viewpoint, is required to develop a useful
conceptualisation of engagement in this context [17].
Engagement has traditionally been conceptualised
differently across the behavioural science, computer
science andHCI literatures, which might be dueto the
different epistemologies subscribed to, the differing
research contexts and the different objectives pursued.
In the computer science and HCI literatures, engage-
ment has traditionally been conceptualised as the sub-
jective experience of flow, a mental state characterised
by focused attention and enjoyment [18]. This kind of
conceptualisation might have emerged as a result of
the focus on entertainment and usability of interactive
technology. In the behavioural science literature, en-
gagement has typically been conceptualised as
Busage^of DBCIs, focusing on the temporal patterns
(e.g. frequency, duration) and depth (e.g. use of specific
intervention content) of usage [19,20]. This kind of
conceptualisation has emerged due to the observation
that while many download and try DBCIs, sustained
usage is typically low [2124]. Henceforth, two work-
ing definitions of engagement as used in the computer
science and HCI literatures (Bengagement as flow^)
and the behavioural science literature (Bengagement
as usage^) are used to scope the space within which
this review is conducted.
Although existing systematic reviews have assessed
whether particular DBCI features (e.g. tailoring,
reminders) are associated with higher engagement
[25,26] and whether engagement is associated with
intervention effectiveness [11], it is not possible to
synthesise results from these reviews or to draw any
conclusions regarding the shape of the function (e.g.
linear, non-linear) relating engagement with interven-
tion outcomes due to the use of incomparable defini-
tions of engagement [11]. In order to reduce fragmen-
tation of research efforts, it would be useful to develop
a shared understanding of how to conceptualise and
operationalise engagement with DBCIs.
A conceptual framework can been defined as Ba
system of concepts, assumptions, and expectations,
and the presumed relationships among them^[27].
Previous conceptual frameworks of engagement have
proposed multiple interacting factors (e.g. social sup-
port, sensory appeal, ease of use) that influence
Bengagement as flow^or Bengagement as usage^
[2830]; however, these frameworks are either not
derived from empirical observations or draw only on
literature from one of many interrelated scientific dis-
ciplines. For example, the framework proposed by
OBrien and Toms [28], notwithstanding its grounding
in empirical observations, drew only on research from
the technology literature and focused on Bengagement
as flow^without any links to behaviour change. Con-
versely, the framework by Ritterband and colleagues
[29]focusedonBengagement as usage^and was de-
rived from behavioural science theory only. The mod-
el proposed by Short and colleagues [30] attempted to
integrate both theoretical predictions and empirical
findings from the behavioural science, persuasive de-
sign and technology literatures but did not do so in a
systematic manner. Although the ontology of behav-
iour change interventions proposed by West and
Michie provides a starting point for organising and
representing DBCIs, engagement constitutes one of
many important components and is hence not exam-
ined in detail [4]. It is therefore not possible to deter-
mine whether existing frameworks of engagement suf-
ficiently explain real-world events, or whether impor-
tant aspects are missing.
The aims of this review are threefold; the second
and third build on output from the first:
1. To synthesise past work on engagement, addressing
the following research questions:
(a) How has engagement been defined in the selected
literatures?
(b) How has engagement been measured?
(c) What factors have been found or hypothesised to
influence engagement?
(d) What are the proposed relationships between en-
gagement and intervention effectiveness?
2. To develop an integrative definition of engagement
with DBCIs and specify how it can be measured.
3. To develop a conceptual framework of the direct
and indirect influences on engagement with DBCIs
and the proposed relationships between engage-
ment and intervention effectiveness.
METHODS
The Cochrane Handbook of Systematic Reviews of Interven-
tions [15] and the Guidance for Undertaking Reviews in
Health Care [31] were used to inform the development
of the search strategy, identify inclusion criteria, select
studies and extract the data. Principles from critical
interpretive synthesis (CIS) were used to inform the
data synthesis [32]. As CIS is one of the few methods
available that affords the synthesis of qualitative and
quantitative data, it was deemed to be the most suitable
method. CIS is useful when a review seeks to identify a
definition of a phenomenon, as it aims to produce a
higher-order structure or conceptual framework
(Bsynthesising argument^), which is grounded in the
SYSTEMATIC REVIEWS
TBMpage 2 of 14
concepts (Bsynthetic constructs^)identifiedinthe
reviewed articles [32]. CIS does not propose a formal
method for critically appraising the quality and meth-
odological rigour of included studies but recognises
that the critical evaluation and integration of disparate
forms of evidence is essentially a product of the
Bauthorial voice^[33]. The evidence is critiqued on
the basis of the implicit assumptions underlying the
methodological decisions made in the reviewed
articles. Hence, the quality of the evidence is consid-
ered in the development of the synthetic constructs,
with the consideration based on the authorsjudge-
ments. Principles of CIS have previously been
employed in reviews of the health literature [3436].
Criteria for considering studies for this review
All types of study designs were included except posi-
tion papers. All types of information sources were
included except articles that were not peer-reviewed
or not available in English. Studies with adult partic-
ipants (i.e. aged 18 years or older) were included, as it
was expected that different factors might influence
engagement in children and adult populations due to
different cognitive abilities [37]. Studies specifically
targeting participants with cognitive impairment or
intellectual disabilities were excluded for the same
reason. DBCIs and digital interventions targeting indi-
viduals with mental health or chronic physical health
conditions were included as no a priori reason suggest-
ing that engagement should be conceptualised differ-
ently across the included topic areas could be identi-
fied. Interventions were excluded if they did not incor-
porate any digital component as part of the interven-
tion itself (i.e. face-to-face delivery only) or if the tech-
nology was used solely as a tool to deliver measure-
ment surveys. Studies involving recreational or educa-
tional digital games or multimedia software (e.g. soft-
ware involving animations, sound and text) were in-
cluded providing that engagement was discussed or
measured. For the conceptualisation of Bengagement
as flow^, the games or technology did not need to be
related to behaviour change. The primary outcome
was definitions of engagement with DBCIs, digital
games or multimedia software expressed either implic-
itly or explicitly. Secondary outcomes included pro-
posed direct and indirect influences on engagement,
measures of engagement and associations between
engagement and intervention effectiveness expressed
either implicitly or explicitly.
Search methods for the identification of studies
Electronic searches
A structured search of the following electronic databases
was conducted in November 2015: Ovid MEDLINE
(1946November 2015), PsycINFO (1806November
2015), ISI Web of Knowledge (1900November 2015)
and ScienceDirect (1900November 2015). Search
terms were piloted and refined to achieve a balance
between sensitivity, i.e. retrieving a high proportion of
relevant articles, and specificity, i.e. retrieving a low
proportion of irrelevant articles [15]. An academic li-
brarian was consulted for the validation of the databases
and the final search terms. Terms were searched for in
titles and abstracts as free text terms or as index terms
(e.g. Medical Subject Headings) where appropriate (see
Electronic Supplementary Material 1).
Searching for other resources
Articles from adjacent fields not immediately or obvi-
ously relevant to the research questions were identified
through expertise within the review team [32]. The
Association for Computing Machinery Digital Library
(a repository for conference proceedings) and relevant
journals (i.e. Journal of Medical Internet Research,Journal
of the American Medical Informatics Association,Telemedi-
cine and e-Health) were hand searched, and reference
chaining was employed to identify additional articles
of interest [15,32].
Data collection and analysis
Selection of studies
Articles identified through the electronic and hand
searches were merged using EndNote X7 [38]toensure
consistency. Duplicate records were removed. Two
researchers independently screened (i) titles, (ii) abstracts
and (iii) full texts of the identified articles against the pre-
defined eligibility criteria [15]. Any disagreements were
resolved through discussion and by consulting a third
researcher if necessary. Inter-rater reliability was
assessed based on two coding categories (i.e. inclusion
versus exclusion) after the full text screening phase with
the prevalence- and bias-adjusted kappa (PABAK) sta-
tistic, which controls for chance agreement [39]. The
following cutoffs were used: 0.400.59 indicates fair
agreement, 0.600.74 indicates good agreement and
>0.75 indicates high agreement [15].
Data extraction and management
A pro-forma was developed by the first author to
extract information about the study setting, participant
characteristics, study design, data collection method
and study findings [32]. The pro-forma was piloted
on a sample of included articles to ensure that relevant
information was captured [15]. A second researcher
independently checked the pro-forma for accuracy
and completeness [31]. Due to limited resources, a
single reviewer completed the data extraction.
Quality appraisal
CIS suggests the prioritisation of seemingly relevant
articles rather than favouring particular study methodol-
ogies [40]. Judgements about the relevance and underly-
ing assumptions of articles were made by the first author
and were incorporated into the data synthesis [32].
SYSTEMATIC REVIEWS
TBM page 3 of 14
Data synthesis
Based on the principles from CIS, the data synthesis
comprised the following steps:
1. Concepts identified in the full texts of included
articles were labelled with codes by the first author.
The research questions were used as a top-down
coding frame; fragments of text explicitly or implic-
itly referring to definitions of engagement, meas-
ures of engagement, influences on engagement or
associations between engagement and intervention
effectiveness were coded.
2. A subsample of codes was selected through random
sequence generation (https://www.random.org/)
for validation by an independent researcher to in-
crease rigour [41]. Disagreements were discussed
until consensus was reached.
3. Synthetic constructs (i.e. concepts that explain sim-
ilar themes) were developed from the codes, and
relationships between synthetic constructs were
specified by the first author.
4. The synthetic constructs and the proposed relation-
ships between constructs were validated by an in-
dependent researcher. Disagreements were dis-
cussed until consensus was reached.
5. Two synthesising arguments (i.e. an integrative def-
inition and its measurement, and a conceptual
framework) were developed based on the synthetic
constructs by the first author.
6. The synthesising arguments were refined through
discussion between all co-authors.
Results
Summary of search results
The electronic database search yielded 925 published
articles. After removing duplicates, 560 articles
remained for screening. A PABAK score of 0.88 was
achieved after the full text screening phase, indicating
high inter-rater reliability [15]. Due to this reliability
score, the additional 31 information sources were
screened by a single reviewer. Of the 140 full texts
screened, 117 met the inclusion criteria and were in-
cluded in the data synthesis. Six qualitative studies, 27
reviews, 2 mixed methods studies and 82 quantitative
studies were included (see Fig. 1). Characteristics of the
included studies are described in Electronic Supple-
mentary Material 2.
Records identified through
database searching
(n = 925)
Screening Included Eligibility Identification
Records after duplicates removed
(n = 560)
Titles screened
(n = 560)
Titles excluded
(n = 292)
Full-text articles
assessed for eligibility
(n = 109)
Full-text articles excluded (n = 23),
with reasons:
Book chapter (n = 5)
Not adults (n = 4)
Meeting abstract/poster (n = 3)
No definition of engagement (n = 6)
Focusing on learning (n = 4)
Not available through library
resources (n = 1)
!
Studies included in qualitative
synthesis
(n = 117)
Abstracts screened
(n = 268)
Abstracts excluded
(n = 159)
Additional records identified
through other sources
(n = 31)
Fig 1 | PRISMA flow diagram of the study selection process [42]
SYSTEMATIC REVIEWS
TBMpage 4 of 14
How has engagement been defined in the literature?
The following two synthetic constructs were devel-
oped: Bengagement as subjective experience^and
Bengagement as behaviour^.
Engagement as subjective experience
Engagement has been conceptualised as the subjective
experience that emerges in the momentary interaction
with a system [18,28,43]. This kind of conceptualisa-
tion was only identified in the computer science and
HCI literatures. Similarities can be found between
engagement and the state of Bflow^, described as a
mental state characterised by focused attention, intrin-
sic interest and enjoyment, balance between challenge
and skill, and temporal dissociation (i.e. losing track of
the passage of time) [18,4447]. Similarities can also
be found between engagement and the state of
Bimmersion^within digital gaming, characterised by
cognitive absorption, the willingness to direct emo-
tions towards an activity and feeling cutoff from reality
[43,4851]. As conceptual overlap was observed be-
tween these experiential qualities, the authors propose
that they can be grouped under the following cognitive
and emotional states: attention, interest and affect.
Engagement as behaviour
The majority of articles reviewed from the behav-
ioural science literature conceptualised engage-
ment in behavioural terms, suggesting that it is
identical to the usage of a DBCI or its compo-
nents. Engagement has further been described as
theextentofusageovertime[19,52], sometimes
referred to as the Bdose^obtained by participants
or Badherence^to an intervention [25,53,54],
determined by assessing the following subdimen-
sions: Bamount^or Bbreadth^(i.e. the total length
of each intervention contact), Bduration^(i.e. the
period of time over which participants are ex-
posed to an intervention), Bfrequency^(i.e. how
often contact is made with the intervention over
a specified period of time) and Bdepth^(i.e. vari-
ety of content used) [20,53]. In the computer
science and HCI literatures, engagement has
been conceptualised as the degree of involvement
over a longer period of time [55], sometimes
referred to as Bstickiness^[56]. A distinction has
also been made between Bactive^and Bpassive^
engagement; while the former involves contribut-
ing to the intervention through posting in an
online discussion forum, the latter involves read-
ing what others have written without comment-
ing, also known as Blurking^[57]. Engagement
has also been conceptualised as a process of
linked behaviours, suggesting that users move
dynamically between stages of engagement, dis-
engagement and re-engagement [28]. As concep-
tual overlap was observed between these defini-
tions, the authors propose that engagement
involves different levels of usage over time.
Development of an integrative definition of engagement
An integrative definition of engagement with DBCIs
was developed through the merging of overlapping
conceptualisations as outlined above, in addition to
the integration of the two overarching synthetic con-
structs. The following two-part definition is therefore
proposed:
BEngagement with DBCIs is (1) the extent (e.g. amount,
frequency, duration, depth) of usage and (2) a subjective
experience characterised by attention, interest and affect^.
Engagement is conceptualised as a multidimension-
al construct: the behavioural dimensions of engage-
ment are underpinned by the users subjective experi-
ence of what it feels like to be engaged with a DBCI.
Engagement is considered to be a dynamic process
that is expected to vary both within and across indi-
viduals over time.
How has engagement been measured?
The following two synthetic constructs were devel-
oped: Bsubjective measures^and Bobjective
measures^.
Subjective measures
In research settings, self-report questionnaires have
frequently been used to measure engagement with
digital games and DBCIs [51,5867]. Qualitative
approaches, such as interviews or think aloud meth-
odology, have been employed to gain a better under-
standing of the nature of usersexperiences of engage-
ment with digital games and DBCIs [60,68,69].
Objective measures
Automatic tracking of use patterns, including number
of logins, time spent online and the amount and type of
content used during the intervention period, was the
most commonly used measure of engagement in the
behavioural science literature [11,19,20,26,44,70
82]. Physiological measures including cardiac activity,
respiratory depth [62] and electro-dermal activity [65],
and psychophysical measures, such as eye tracking
[51], have been used to measure engagement in the
computer science and HCI literatures.
Measures relating to the integrated conceptualisation of
engagement
Based on the literature synthesis, we suggest that all
facets of engagement proposed in the integrative def-
inition of engagement can in principle be measured or
inferred through the following: (1) user-reported inter-
action with the DBCI through self-report question-
naires, interview studies or think aloud studies; (2)
automated recording of DBCI use (e.g. logins, page
views); and (3) recording of physiological or psycho-
physical correlates of DBCI interaction.
SYSTEMATIC REVIEWS
TBM page 5 of 14
What factors have been hypothesised or found to influence
engagement?
The following two synthetic constructs were devel-
oped: Bcontext^and BDBCI^. Context was subdi-
vided into Bpopulation^and Bsetting.^DBCI was sub-
divided into Bcontent^and Bdelivery.^Relationships
between constructs were specified.
Context
Population
Psychological characteristicsMotivation was found to be
positively associated with engagement across many
studies, with none indicating a negative association
[20,68,8387]. As the available evidence is correla-
tional in nature, the direction of influence cannot be
assumed. It has been hypothesised that the relation-
ship between motivation and engagement might be U-
shaped; those who are least and most motivated to, for
example, quit smoking, are hypothesised to disengage
quickly from DBCIs due to failed and successful be-
haviour change, respectively [19].
Expectations are thought to be influential in that users
are hypothesised to engage more if there is a match
between their expectations and the goal of the DBCI
[49,73,86,88,89]. Prior experiences of using other
websites or apps, or of having tried face-to-face
counselling (which may or may not have worked),
might shape usersexpectations of what DBCIs can
provide [90].
Mental health, including low mood, anxiety and stress,
has been found to be negatively associated with en-
gagement [68,73,87,9196]. A negative association
with mental health was mainly observed in studies of
DBCIs targeting individuals diagnosed with a mental
health condition but was also observed in physical
activity [68] and weight loss [94]interventions.Simi-
larly, experience of well-being or believing that one does
not need to work on certain issues has been found to
be negatively associated with engagement [92].
Need for cognition, defined as the tendency to process
large amounts of information [11,30,57,88,97], and
self-efficacy to execute a given behaviour [83,98,99]
were found to be positively associated with
engagement.
Personal relevance, which refers to the extent to
which a DBCI is perceived to apply to the indi-
vidual and their particular situation, has been
hypothesised to positively influence engagement
[69,78,100104]. Results from interview studies
indicate that participants believe that lack of per-
sonal relevance is a sufficient reason for dropping
out from intervention trials [86,92,95,105].
Demographic characteristicsAge [20,57,63,6870,73,
76,79,91,95,96,99,106111], gender [20,69,73,90,
95,100,101,110,111], education [20,69,91,92,96,99,
106,107,109,110,112], employment [91,92,107]and
ethnicity [57,106]were found to be significantly asso-
ciated with engagement. There was a trend towards a
positive association between engagement and older
age, higher educational attainment and being a wom-
an; however, as no meta-analysis was conducted, a
conclusion about the size and direction of influence
cannot be drawn. Computer literacy, or confidence using
the Internet, has been found to be positively associated
with engagement [11,20,98,99,106,108,113]. How-
ever, as none of the included studies adequately mea-
sured baseline computer skills in their designs, a firm
conclusion cannot be drawn.
Physical characteristicsPhysique, including baseline
weight and the presence of comorbidities, was found
to be negatively associated with engagement [68,70,
71,9194,106,112].
Setting
The social and physical environments in which a DBCI
is used, have been hypothesised to influence engage-
ment [4,29,30]. The social environment includes
culture (e.g. prevailing norms), commercial environ-
ment, media and social cues. The physical environ-
ment includes financial resources, material resources,
time pressure, physical cues, location, the healthcare
system and policy. Time [86,92,93,114]andaccess to
hardware or the Internet [30,115] have been hypoth-
esised to be positively associated with engagement.
DBCI
Content
DBCIs that include particular behaviour change techni-
ques (BCTs), such as action plans [78], goal setting
[116], feedback [59] and self-monitoring tools [78],
have been found to be associated with higher engage-
ment [78]. Rewards and incentives have been hypothes-
ised [26,100,101,117] or found [118] to positively
influence engagement; however, evidence from trials
in which the presence of rewards or incentives has
been manipulated is scarce.
Social support features, referring to features that facil-
itate the receipt of social support, were found to posi-
tively influence engagement [76,82,119124]. Fea-
tures that decrease the feeling of loneliness or that
increase motivation through competition with others
include online discussion forums, gamification ele-
ments such asleaderboards that show users where they
rank in a gamified system, and peer-to-peer contact
[125,126]. Evidence indicates that DBCIs that provide
access to such features are successful in getting users
who report lower social support at baseline to engage
[57,127]; however, participants who reported higher
levels of social support at baseline were found to be
more likely to engage with the social elements of
DBCIs across a few studies [68,86,91,96].
Reminders have been hypothesised [117,128,129]or
found to positively influence engagement; results from
a meta-analysis indicate a positive effect of reminders
on engagement [130]. However, receiving too many
SYSTEMATIC REVIEWS
TBMpage 6 of 14
reminders may have a negative effect on engagement
due to Be-mail fatigue^[69].
Delivery
Mode of delivery, which includes face-to-face, telephone,
text message, smartphone app, website and mass me-
dia delivery, has been hypothesised to influence en-
gagement with DBCIs [4].
Professional support features, which include features
that enable remote contact with a clinician via e-mail,
telephone or text messages, have been found to posi-
tively influence engagement with DBCIs [20,25,26,
63,68,70,73,77,88,90,95,120,131134]. However,
results from a randomised controlled trial (RCT) of a
web-based weight loss intervention in which some
participants received coaching calls from a nurse indi-
cated that participants in the coaching arm were more
likely to drop out around the time of the first coaching
session, suggesting a negative influence of professional
support features in particular situations [70].
Control features, referring to features that make users
feel that they are in control of and are free to make
choices about how to interact with a DBCI, have been
hypothesised [51,119]orfound[52,74,110]toposi-
tively influence engagement. For example, results
from an RCT in which participants either received
content all at once or sequentially over a period of
weeks suggest that participants were more likely to
disengage when the content was delivered sequentially
[110]. Tunnelled interventions (i.e. those that lead
users through a number of predetermined steps) have
been found to generate more page views compared
with self-paced ones [74]. However, this may be an
artefact of making users click through a pre-specified
number of pages in order to progress through the
DBCI.
Novelty, generated by regular content updates, has
been found to positively influence engagement
through preventing boredom [25,26]. However, there
might be a trade-off between novelty and programme
complexity; it has been hypothesised that participants
will disengage if the intervention is perceived as too
long or overly complicated [26,68,73,88,131,135,
136]. It has been hypothesised that the presence
of too many features may compromise a DBCIs
ease of use [19], referring to whether or not it feels
natural for the user to operate an interactive
system. Ease of use has been hypothesised to
positively influence engagement [71,100,137].
The personalisation or tailoring of content has been
hypothesised [26,52,68,72,80,103,106,110,113,
119,120,138] or found [19,20,66] to positively
influence engagement. Interactivity, referring to a two-
way flow of information between a DBCI and its user,
has been hypothesised [28,48,50,66,78,100,139]or
found [19] to positively influence engagement.
Message tone, which refers to the terminology and
wording used to communicate health messages [92,
101], and narrative [43,50,65,103,125,140], referring
to the presence of a storyline, have been hypothesised
to positively influence engagement. Furthermore, chal-
lenge [61,100,141], aesthetics and design [120,139,142,
143]andcredibility features [68,73], referring to features
that inculcate a feeling of trust, familiarity [97,139,
144], and the provision of guidance or tutorials [68,
106,145] have been hypothesised to positively influ-
ence engagement with DBCIs.
What are the proposed relationships between engagement
and the effectiveness of DBCIs?
The following four synthetic constructs were devel-
oped to explain the proposed relationships between
engagement and the effectiveness of DBCIs:
Bmechanisms of action^,Bunmeasured third variable^,
Boptimal dose^and Beffective features^.
Mechanisms of action
Mechanisms of action proposed to mediate the effect of
engagement with DBCIs on intervention effectiveness
[4] include increased knowledge, motivation, affect
management, cognitive restructuring, skill building
[29], comprehension and practice of programme con-
tent, and increased self-efficacy [19]. A further distinc-
tion has been made between Bintervention receipt^,
which refers to the extent to which participants under-
stand and can perform the skills taught, and
Benactment of intervention skills^, which refers to the
extent to which participants use these skills [146,147].
It has also been hypothesised that mechanisms of
action, such as accountability to a healthcare practi-
tioner and relatedness to other individuals, might pos-
itively influence engagement with DBCIs [68,77,86,
96].
Unmeasured third variable
An unmeasured third variable, such as higher baseline
motivation or self-efficacy, may be responsible for the
observed association between increased engagement
and positive DBCI outcomes. Alternatively, those who
engage with DBCIs might simply be more inclined to
behave healthily in general [11]. It has also been ar-
gued that the target behaviour itself might influence
engagement [148]. For example, smokers who relapse
might be more likely to stop engaging with the DBCI,
while those who successfully manage their cravings
might be more likely to continue engaging with the
DBCI.
Optimal dose
Optimal dose refers to a pre-defined level of engage-
ment at which specific DBCIs are effective. It has been
hypothesised that the receipt of an optimal dose may
explain the relationship between engagement and in-
tervention effectiveness but that the optimal dose for
particular DBCIs may vary depending on user char-
acteristics [70,113].
SYSTEMATIC REVIEWS
TBM page 7 of 14
Effective features
The use of specific intervention features has been
found to be associated with better DBCI outcomes
[70]. It has been suggested that there may be a mis-
match between features that participants choose to
engage with frequently and effective features that are
causally linked to intervention outcomes [104]. For
example, although users may enjoy engaging with a
particular feature (e.g. filling out a food diary), thus
using it frequently, use of a less popular feature (e.g.
Bgetting support^tools) might be more strongly asso-
ciated with intervention outcomes, such as weight
loss [70].
Development of a conceptual framework of engagement with
DBCIs
The final aim of the review was to develop a concep-
tual framework specifying potential direct and indirect
influences on engagement and relationships between
engagement and intervention effectiveness. As the
framework proposed by Ritterband and colleagues
[29] and the ontology proposed by West and Michie
[4] explicitly linked engagement to behaviour change,
we drew on these to structure our conceptual frame-
work, mapping the other existing frameworks onto it.
Additional factors identified in the reviewed literature
not otherwise specified were also mapped onto the
conceptual framework.
We propose a conceptual framework in which en-
gagement with a DBCI influencesthe target behaviour
through specific mechanisms of action; box 4, box 1,
box 3 and box 2, respectively. Content has been found
to directly influence engagement with DBCIs; box a.
Delivery has been hypothesised to directly influence
engagement with DBCIs; box b. The context and the
target behaviour are hypothesised to directly influence
engagement; box 5 and box 3, respectively. Mecha-
nisms of action are hypothesised to indirectly influ-
ence engagement; box 2. The population (e.g. demo-
graphic, physical and psychological characteristics)
has been found to directly influence engagement with
DBCIs; box c. The setting has been hypothesised to
directly influence engagement; box d. Engagement is
hypothesised to be indirectly influenced by the mod-
erating influence of the context on the influence of the
DBCI; box 4, box 5 and box 1, respectively. Figure 2
shows this schematically. Hypothesised influences are
marked with stars.
DISCUSSION
An integrative conceptualisation of engagement with
DBCIs has been developed; engagement is defined
here as a multidimensional construct which can be
measured through self-report questionnaires, verbal
reports, automatic recording of DBCI use or recording
of psychophysical manifestations. A conceptual frame-
work was developed, which suggests that the context
of use influences engagement with DBCIs either di-
rectly or indirectly by moderating the influence of the
DBCI on engagement. Mechanisms of action might
indirectly influence engagement and the target
Fig 2 | Conceptual framework of direct and indirect influences on engagement with DBCIs. Transparent boxes indicate concepts.
Concepts can be defined as abstract ideas that are derived from either direct or indirect evidence [149]. Blue boxes indicate
attributes of concepts. Attributes can be defined as properties that characterise a concept [150]. Solid black arrows indicate
relationships between concepts and attributes. Arrows with transparent heads indicate an influence of a concept.
SYSTEMATIC REVIEWS
TBMpage 8 of 14
behaviour might directly influence engagement with
DBCIs, suggesting the presence of a positive feedback
loop. The proposed relationships between engage-
ment and intervention effectiveness are tentative, as
these have not been studied extensively.
The suggested behavioural and experiential dimen-
sions of engagement can in principle be measured or
inferred in every instance of a DBCI. The content,
structure, length and design of specific DBCIs tend to
vary, and hence, the relevance of the different dimen-
sions of engagement will vary accordingly. Although the
intended frequency, amount, duration and depth of use
might be set to B1^in a one-off intervention, the indi-
vidual parameters are still present and measureable.
Thus, the proposed definition of engagement allows
for direct comparison across different kinds of DBCIs
by including multiple dimensions of engagement at its
core. This has been lacking in previous conceptualisa-
tions. Evidence of higher engagement coupled with
evidence of, for example, enjoyment of using a DBCI
is hypothesised to predict greater DBCI effectiveness. If
this is the case, the proposed definition of engagement
should provide a means of generalising findings from
particular DBCIs to other similar DBCIs. It may not be
possible to evaluate the usefulness of the proposed def-
inition prior to empirical work [151].
Although some self-report questionnaires designed
to measure engagement demonstrate good validity
and reliability [64,152], these typically rely on measur-
ing engagement after, as opposed to during, the event.
However, the advent of new technologies allows self-
reports of engagement to be measured in real-time
rather than through paper-and-pencil questionnaires
[153]. Although physiological measures have been used
to measure engagement, notably in the HCI literature,
associations between physiological and self-reported
measures of engagement are weak [65].Thenatureof
these associations should be investigated further.
Previous conceptual frameworks have been based on
theoretical predictions only or have been derived from
the literature within one scientific domain [4,2830]. In
contrast, our conceptual framework is derived from the-
oretical predictions and empirical observations within
multiple, interrelated disciplines. This endeavour was
facilitated by the use of principles from CIS, which
allowed the combination of a diverse set of research
findings. The proposed conceptual framework of engage-
ment is a synthesis of existing ontologies, frameworks
and models and incorporates factors not previously in-
cluded. The novel components in our framework are as
follows: Bmental health^,Bexperience of well-being^,
Bfamiliarity^,Bguidance^and Bnarrative^. The negative
association between poor mental health and engagement
might be explained by the observation that those with
poor mental health (e.g. depression) typically experience
decreased self-efficacy to, for example, stop smoking or
lose weight [154,155]. Experience of well-being might be
negatively associated with engagement due to being re-
lated to the belief that one does not need any support.
Familiarity with the design of DBCIs and guidance
might positively influence engagement because
familiar examples, design conventions or stepped how-
to-use guides may inculcate feelings of comfort and ease
of use. A narrative might draw users in, increasing their
interest and enjoyment. Moreover, this review identi-
fied a trend towards a positive association between
engagement and older age, higher educational attain-
ment and being a woman, which merits further inves-
tigation. Although these demographic characteristics
have been included in existing frameworks of engage-
ment, the direction of influence has not been previous-
ly discussed. Through the use of a systematic, interdis-
ciplinary approach, the proposed conceptual frame-
work offers a comprehensive overview of the factors
that may influence engagement with DBCIs and hence
provides a starting point for reducing the observed
fragmentation of research findings.
LIMITATIONS
The lack of evidence supporting the claim that setting
of use (e.g. culture, social norms, physical cues, health-
care pathway) directly influences engagement with
DBCIs constitutes a limitation. This might either re-
flect the search terms used or indicate that this has not
been investigated in the literature; we cannot distin-
guish between these explanations. There was also a
lack of evidence in support of the claim that the con-
text of use (i.e. setting and population) may moderate
the influence of the DBCI on engagement. For exam-
ple, the setting of use may vary depending on the
mode of delivery (e.g. computer versus mobile phone).
Hence, the DBCI might indirectly influence engage-
ment through determining the setting of use; while
computers may predominantly be used at home or in
a clinic, mobile phones might mainly be used on the
go, which may influence the amount or depth of en-
gagement. Future research should test this hypothesis.
Another limitation is that no formal quality assessment
of the included articles was conducted. However, this
was in line with the chosen method, which suggests
that the articles should be judged on the basis of their
relevance to the research question rather than their
methodological rigour. This method was selected due
to the conceptual nature of the research questions. A
further limitation is that the data extraction and litera-
ture synthesis were conducted by a single reviewer,
potentially introducing bias. Finally, the end date for
the literature search (i.e. November 2015) constitutes a
limitation; with the pace of technological advances and
the proliferation of digital health research, it is likely
that relevant literature has since been published.
IMPLICATIONS AND AVENUES FOR FUTURE RESEARCH
The proposed integrative definition and conceptual
framework of engagement with DBCIs have implica-
tions for clinical practice: the use of a shared terminology
and measurement techniques will ensure more rapid
advance in understanding engagement with DBCIs
and developing methods to improve it. A shared con-
ceptualisation of engagement can be used to help
SYSTEMATIC REVIEWS
TBM page 9 of 14
policymakers and commissioners to set evaluation
standards for DBCIs. Moreover, the proposed concep-
tual framework can be used to generate testable hypoth-
eses about how to improve engagement with DBCIs.
For example, according to the conceptual framework,
the presence of rewards might influence engagement
with a DBCI due to increased motivation. This hypoth-
esised link between rewards, motivation and engage-
ment can be tested using an experimental design. Future
avenues for research include the assessment of what
dimensions of engagement (e.g. attention, interest, affect,
amount, duration, frequency, depth) are most strongly
associated with intervention effectiveness, whether it is
possible to establish benchmarks for the optimal dose of
engagement across different kinds of DBCIs and wheth-
er the context of use influences engagement with DBCIs.
CONCLUSION
Engagement with DBCIs is conceptualised here in terms
of both experience and behaviour. Engagement may be
influenced by the DBCI itself, the context of use, mech-
anisms of action of the DBCI and the target behaviour.
Acknowledgements: The authors would like to thank Jacqui Smith, librarian
at University College London, for helping to build the search strategy; Nicola
Newhouse for validating the coding frame and commenting on an early draft;
and Holly Walton for helping to screen articles. Olga Perski is a Ph.D. candidate
funded by a grant from Bupa under its partnership with University College
London. Susan Michie is part funded by grants from Cancer Research UK and
NIHRs School for Public Health Research. Robert West is funded by Cancer
Research UK. The funders played no role in the design, conduct or analysis of
the study nor in the interpretation and reporting of study findings.
Compliance with ethical standardsStatements on human rights, the
welfare of animals, informed consent, and the Declaration of Helsinki are not
applicable to this manuscript. IRB approval is not applicable.
Ethical responsibilities of authors: All authors have approved the final
manuscript and agree with its submission to Translational Behavioural Medi-
cine.All authors have contributed equally to the scientific work and are
responsible and accountable for the results. We confirm that this manuscript
has not been previously published (partly or in full) and that the manuscript is
not being simultaneously submitted elsewhere.We confirm thatthe data have
not been previously reported elsewhere and that no data have been fabricated
or manipulated to support our conclusions. No data, text or theories by others
are presented as if they were the authorsown.The authors have full control of
all data, which are accessible upon request.
Conflict of interest: OP, SM and AB declare that they have no conflict of
interest. RW undertakes research and consultancy and receives fees for speaking
from companies that develop and manufacture smoking cessation medications.
Open Access This article is distributed under the terms of the Creative
Commons Attribution 4.0 International License (http://creativecommons.org/
licenses/by/4.0/),which permitsunrestricted use, distribution,and reproduction
in any medium, provided you give appropriate credit to the original author(s)
and the source, provide a link to the Creative Commons license, and indicate if
changes were made.
1. Rock Health. (2015). Digital health consumer adoption: 2015.
Retrieved November 4, 2015, from http://rockhealth.com/
reports/digital-health-consumer-adoption-2015/.
2. Fox S, Duggan M. Mobile health 2012. Pew Internet & American
Life Project. 2012 Retrieved from http://www.pewinternet.org/
2012/11/08/mobile-health-2012/.
3. Kontos, E., Blake, K. D., Chou, W.-Y. S., & Prestin, A. (2014).
Predictors of eHealth usage: insights on the digital divide from
the Health Information National Trends Survey 2012. Journal of
Medical Internet Research, 16(7), e172. doi:10.2196/jmir.3117.
4. West, R., & Michie, S. (2016). A Guide to Development and Evalu-
ation of Digital Interventions in Healthcare. London: Silverback
Publishing.
5. Civljak, M., Stead, L. F., Sheikh, A., & Car, J. (2013). Internet-
based interventions for smoking cessation. Cochrane Database
Syst Rev, 7,CD007078.
6. Whittaker, R., Borland, R., Bullen, C., Rb, L., Mcrobbie, H., &
Rodgers, A. (2009). Mobile phone-basedinterventions for smok-
ing cessation. Cochrane Database Syst Rev, 4.
7. Nair, N. K.,Newton, N. C., Shakeshaft, A., Wallace, P.,& Teesson,
M. (2015). A systematic review of digital and computer-based
alcoholintervention programs in primary care. Current Drug Abuse
Reviews, 8(2), 111118.
8. Liu, F., Kong, X., Cao, J., Chen, S., Li, C., Huang, J., et al. (2015).
Mobile phone intervention and weight loss among overweight
and obese adults: a meta-analysis of randomized controlled
trials. Am J Epidemiol, 181(5), 337348. doi:10.1093/aje/
kwu260.
9. Muntaner, A., Vidal-Conti, J., & Palou, P. (2015). Increasing
physical activity through mobile device interventions: a system-
atic review. Health Informatics Journal,119. doi:10.1177/
1460458214567004.
10. Jones, K. R., Lekhak, N., & Kaewluang, N. (2014). Using mobile
phones and short message service to deliver self-management
interventions for chronic conditions: a meta-review. Wo r l dv i e ws
on Evidence-Based Nursing/Sigma Theta Tau International, Honor
Society of Nursing, 11(2), 8188. doi:10.1111/wvn.12030.
11. Donkin, L., Christensen, H.,Naismith, S. L., Neal, B.,Hickie, I. B.,
& Glozier, N. (2011). A systematic review of the impact of
adherence on the effectiveness of e-therapies. Journal of Medical
Internet Research, 13(3), e52. doi:10.2196/jmir.1772.
12. Cobb, N. K., Graham, A. L., Bock, B. C., Papandonatos, G., &
Abrams, D. B. (2005). Initial evaluation of a real-world Internet
smoking cessation system. Nicotine & Tobacco Research, 7(2),
207216. doi:10.1080/14622200500055319.
13. Tate, D. F., Wing, R. R., & Winett, R. a. (2001). Using Internet
technologyto deliver a behavioralweight loss program. JAmMed
Assoc, 285(9), 11721177. doi:10.1001/jama.285.9.1172.
14. Alexander, G. L., McClure, J. B., Calvi, J. H., Divine, G. W., Stop-
poni, M. A., Rolnick, S. J., et al. (2010). A randomized clinical trial
evaluating online interventions to improve fruit and vegetable
consumption. Am J Public Health, 100(2), 319326. doi:10.2105/
AJPH.2008.154468.
15. The Cochrane Collaboration. Cochrane Handbook for Systematic
Reviews of Interventions Version 5.1.0. [Updated March 2011]. (J.
Higgins & S. Green, Eds.) 2011 Retrieved from www.cochrane-
handbook.org.
16. KrishnanA What are academic disciplines? NCRM Working Paper
Series: ESRC National Centre for Research Methods (2009).
17. Krishnan A Five strategies for practising interdisciplinarity. NCRM
Working Paper Series: ESRC National Centre for Research Methods.
(2009). Retrieved from http://eprints.ncrm.ac.uk/782/1/
strategies_for_practising_interdisciplinarity.pdf.
18. Csikszentmihalyi, M. (1990). Flow: the Psychology of Optimal Per-
formance. New York: Cambridge University Press.
19. Danaher, B. G.,Boles, S. M., Akers, L., Gordon, J. S., & Severson,
H. H. (2006). Defining participant exposure measures in web-
based health behavior change programs. Journal of Medical Inter-
net Research, 8(3), e15. doi:10.2196/jmir.8.3.e15.
20. Couper, M. P., Alexander, G. L., Zhang, N., Little, R. J. A., Maddy,
N., Nowak, M. A., et al. (2010). Engagement and retention:
measuring breadth and depth of participant use of an online
intervention. Journal of Medical Internet Research, 12(4), e52.
doi:10.2196/jmir.1430.
21. Eysenbach, G. (2005). The law of attrition. Journal of Medical
Internet Research, 7(1), e11. doi:10.2196/jmir.7.1.e11.
22. Consumer Health Information Corporation. Motivating patients to
use smartphone health apps. (2015). Retrieved August 10, 2015,
from http://www.prweb.com/releases/2011/04/prweb5268884.
htm.
23. Bennett, G. G., & Glasgow, R. E. (2009). The delivery of public
health interventions via the Internet: actualizing their potential.
Annu Rev Public Health, 30, 273292. doi:10.1146/annurev.
publhealth.031308.100235.
24. Brouwer, W., Oenema, A., Raat, H., Crutzen, R., De Nooijer, J., De
Vries, N. K., & Brug, J. (2010). Characteristics of visitors and
revisitors to an Internet-delivered computer-tailored lifestyle in-
tervention implemented for use by the general public. Health
Educ Res, 25(4), 585595. doi:10.1093/her/cyp063.
25. Kelders,S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen,
J. E. W. C. (2012). Persuasive system design does matter: a
systematic review of adherence to web-based interventions.
SYSTEMATIC REVIEWS
TBMpage 10 of 14
Journal of Medical Internet Research, 14(6), e152. doi:10.2196/jmir.
2104.
26. Schubart, J. R., Stuckey, H. L., Ganeshamoorthy, A., & Scia-
manna, C. N. (2011). Chronic health conditions and internet
behavioral interventions: a review of factors to enhance user
engagement. Computers,Informatics,Nursing,29(2), 8192.
doi:10.1097/NCN.0b013e3182065eed.
27. Huberman, M. A., & Miles, M. B. (1994). Data management and
analysis methods. In Handbook of Qualitative Research (pp. 428
443). Thousand Oaks: SAGE Publications.
28. OBrien, H. L., & Toms, E. G. (2008). What is user engagement? A
conceptual framework for defining user engagement with tech-
nology. J Am Soc Inf Sci Technol, 59(6), 938955.
29. Ritterband, L. M., Thorndike, F. P., Cox, D. J., Kovatchev, B. P., &
Gonder-Frederick, L. a. (2009). A behavior change model for
internet interventions. Ann Behav Med, 38,1827. doi:10.1007/
s12160-009-9133-4.
30. Short, C. E., Rebar, A. L., Plotnikoff, R. C., & Vandelanotte, C.
(2015). Designing engaging online behaviour change interven-
tions: a proposed model of user engagement. The European
Health Psychologist, 17(1), 3238.
31. Centre for Reviews and Dissemination, U. of Y. Systematic reviews:
CRDs guidancefor undertaking reviews in healthcare. (K. Khan, G. Ter
Riet, J. Glanville, A. Sowden, & J.Kleijnen, Eds.) (2008). Retrieved
from http://eprints.whiterose.ac.uk/1139/.
32. Dixon-Woods, M., Cavers, D., Agarwal, S., Annandale, E., Arthur, A.,
Harvey, J., et al. (2006). Conducting a critical interpretive synthesis
of the literature on access to healthcare by vulnerable groups. BMC
Med Res Methodol, 6, 35. doi:10.1186/1471-2288-6-35.
33. Dixon-Woods, M., Bonas, S., & Booth, A. (2006). How can system-
atic reviews incorporate qualitative research? A critical perspective.
Qual Res, 6(1), 2744. doi:10.1177/1468794106058867.
34. Entwistle, V., Firnigl, D., Ryan, M., Francis, J., & Kinghorn, P.
(2012). Which experiences of health care delivery matter to
service users and why? A critical interpretive synthesis and
conceptual map. Journal of Health Services Research & Policy,
17(2), 7078. doi:10.1258/jhsrp.2011.011029.
35. Kazimierczak, K. A., Skea, Z. C., Dixon-Woods, M., Entwistle, V.
A., Feldman-Stewart, D., NDow, J. M. O., & MacLennan, S. J.
(2013). Provision of cancer information as a Bsupport for navi-
gating the knowledge landscape^: findings from a critical inter-
pretive literature synthesis. Eur J Oncol Nurs, 17(3), 360369.
doi:10.1016/j.ejon.2012.10.002.
36. Morrison, L., Yardley, L., Powell, J., & Michie, S. (2012). What design
features are used in effective e-health interventions? A review using
techniques from critical interpretive synthesis. Telemedicine and e-
Health, 18(2), 137144. doi:10.1089/tmj.2011.0062.
37. Anderson, P. J. (2002). Assessment and development of execu-
tive function (EF) during childhood. Child Neuropsychology, 8(2),
7182. doi:10.1076/chin.8.2.71.8724.
38. Thomson Reuters. EndNote X7. Philadelphia, USA 2013.
39. Byrt, T., Bishop, J., & Carlin, J. B. (1993). Bias, prevalence and
kappa. J Clin Epidemiol, 46(5), 423429. doi:10.1016/0895-
4356(93)90018-V.
40. Dixon-Woods, M., Sutton, A., Shaw, R., Miller, T., Smith, J.,
Young, B., et al. (2007). Appraising qualitative research for
inclusion in systematic reviews: a quantitative and qualitative
comparisonof three methods. Journalof Health Services Research &
Policy, 12(1), 4247. doi:10.1258/135581907779497486.
41. Barbour, R. S. (2001). Checklists for improving rigour in qualita-
tive research: a case of the tail wagging the dog? Br Med J, 322,
11151117. doi:10.1136/bmj.322.7294.1115.
42. Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Pre-
ferred reporting items for systematic reviews andmeta-analyses:
the PRISM A statement. PLoS Med, 6(7), e1000097. doi:10.1371/
journal.pmed.1000097.
43. Brown E, Cairns P. A grounded investigation of game immersion. In
CHI 04 Extended Abstracts on Human Factors in Computing Systems.
(2004) 12971300. ACM. doi:10.1145/985921.986048.
44. Bianchi-Berthouze N, Kim WW, Patel D. Does body movement
engage you more in digital game play? and why? In Proceedings of
the International Conference on Affective Computing and Intelligent
Interaction. 2007: 102113.
45. Chou JC, Hung C, Hung Y. Design factors of mobile games for
increasinggamersflow experiences. In Proceedings of the 2014 I.E.
ICMIT. 2014:137139.
46. Sharek, D., & Wiebe, E. (2014). Measuring video game engage-
ment through the cognitive and affective dimensions. Simulation
&Gaming,45,569592. doi:10.1177/1046878114554176.
47. Zhou, T. (2013). Understanding the effect of flow on user adop-
tion of mobile games. Personal & Ubiquitous Computing, 17,741
748. doi:10.1007/s00779-012-0613-3.
48. Oh, J., & Sundar, S. S. (2015). How does interactivity persuade?
An experimental test of interactivity on cognitive absorption,
elaboration, and attitudes. J Commun, 65, 213236. doi:10.
1111/jcom.12147.
49. Bouvier, P., Lavoue, E., & Sehaba, K. (2014). Defining engage-
ment and characterizing engaged-behaviors in digital gaming.
Simulation & Gaming, 45(45), 491507. doi:10.1177/
1046878114553571.
50. Schønau-Fog, H., & Bjørner, T. (2012). BSure, I would like to
continue^:a method for mapping the experience of engagement
in video games. Bull Sci Technol Soc, 32(5), 405412. doi:10.
1177/0270467612469068.
51. Jennett,C., Cox, A. L., Cairns, P.,Dhoparee, S., Epps, A., Tijs, T., &
Walton, A. (2008). Measuring and defining the experience of
immersion in games. International Journal of Human-Computer
Studies, 66(9), 641661.
52. McClure,J. B., Shortreed, S. M., Bogart, A., Derry, H.,Riggs, K., St
John, J., et al. (2013). The effect of program design on engage-
ment with an internet-based smoking intervention: randomized
factorial trial. Journal of Medical Internet Research, 15(3), e69.
doi:10.2196/jmir.2508.
53. Voils, C. I., King, H. A., Maciejewski, M. L., Allen, K. D., Yancy Jr.,
W. S., & Shaffer, J. A. (2014). Approaches for informing optimal
dose of behavioral interventions. Ann Behav Med, 48,392401.
doi:10.1007/s12160-014-9618-7.
54. Wang, J., Sereika, S. M., Chasens, E. R., Ewing, L. J., Matthews, J.
T., & Burke, L. E. (2012). Effect of adherence to self-monitoring of
diet and physical activity on weight loss in a technology-
supported behavioral intervention. Patient Preference and Adher-
ence, 6,221226.
55. Calleja, G. (2007). Digital game involvement. Games & Culture,
2(3), 236260.
56. Lin, J. C.-C. (2007). Online stickiness: its antecedents and effect
on purchasing intention. Behav Inform Technol, 26(6), 507516.
doi:10.1080/01449290600740843.
57. Han, J. Y., Kim, J.-H., Yoon, H. J., Shim, M., McTavish, F. M., &
Gustafson, D. H. (2012). Social and psychological determinants
of levels of engagement with an online breast cancer support
group: posters, lurkers, and non-users. JHealthCommun,17(3),
356371. doi:10.1080/10810730.2011.585696.Social.
58. Burns, C. G., & Fairclough, S. H. (2015). Use of auditory event-
related potentials to measure immersion during a computer
game. Int J Hum Comput Stud, 73, 107114. doi:10.1016/j.
ijhcs.2014.09.002.
59. Chiang, Y.-T., Lin, S. S. J., Cheng, C.-Y., & Liu, E. Z.-F. (2011). Exploring
onlinegameplayersflow experiences and positive affect. TheTurkish
Online Journal of Educational Technology, 10(1), 106114.
60. Chung, J., & Gardner, H. J. (2012). Temporal presence variation in
immersive computer games. International Journal of Human-
Computer Interaction, 28(8), 511529. doi:10.1080/10447318.
2011.627298.
61. Fang, X., Zhang, J., & Chan, S. S. (2013). Development of an
instrumentfor studying flow in computergame play. International
Journal of Human-Computer Interaction, 29(7), 456470. doi:10.
1080/10447318.2012.715991.
62. Harmat, L., Manzano, Ö. D., Theorell, T., Högman, L., Fischer, H.,
& Ullén, F. (2015). Physiologicalcorrelates of the flow experience
during computer game playing. Int J Psychophysiol, 97,17.
doi:10.1016/j.ijpsycho.2015.05.001.
63. Hilvert-Bruce, Z., Rossouw, P. J., Wong, N., Sunderland, M., &
Andrews, G. (2012). Adherence as a determinant of effective-
ness of internet cognitive behavioural therapy for anxiety and
depressive disorders. Behav Res Ther, 50(78), 463468. doi:10.
1016/j.brat.2012.04.001.
64. Lefebvre, R. C., Tada, Y., Hilfiker, S. W., & Baur, C. (2010). The
assessment of user engagement with eHealth content: the
eHealth engagement scale. JComput-MediatCommun,15,666
681. doi:10.1111/j.1083-6101.2009.01514.x.
65. Martey, R. M., Kenski, K., Folkestad, J., Feldman, L., Gordis, E.,
Shaw, A., et al. (2014). Measuring game engagement: multiple
methods and construct complexity. Simulation & Gaming, 45,
528547. doi:10.1177/1046878114553575.
66. Morrison, L., Moss-Morris, R., Michie, S., & Yardley, L. (2014).
Optimizing engagement with Internet-based health behaviour
change interventions: comparison of self-assessment with and
without tailored feedback using a mixed methods approach. Br J
Health Psychol, 19,839855. doi:10.1111/bjhp.12083.
67. OBrien, H. L., & Toms, E. G. (2010). The development and
evaluation of a survey to measure user engagement. Journal of
the American Society for Information Science & Technology, 61(1), 50
69. doi:10.1002/asi.
68. Bossen, D., Buskermolen, M., Veenhof, C., de Bakker, D., &
Dekker, J. (2013). Adherence to a web-based physical activity
intervention for patients with knee and/or hip osteoarthritis: a
mixed method study. Journal of Medical Internet Research, 15(10),
e223. doi:10.2196/jmir.2742.
69. Geraghty, A. W. A., Torres, L. D., Leykin, Y. A. N., & Mun, R. F.
(2012). Understanding attrition from international internet
health interventions: a step towards global eHealth. Health Pro-
mot Int, 28(3), 442452. doi:10.1093/heapro/das029.
SYSTEMATIC REVIEWS
TBM page 11 of 14
70. Arden-Close, E. J., Smith,E., Bradbury, K., Morrison, L., Dennison,
L., Michaelides, D., & Yardley, L. (2015). A visualization tool to
analyse usage of web-based interventions: the example of pos-
itive online weight reduction (POWeR). Journal of Medical Internet
Research, 2(1), e8. doi:10.2196/humanfactors.4310.
71. Carter, M. C., Burley, V. J., Nykjaer, C., & Cade, J. E. (2013).
Adherence to a smartphone application for weight loss com-
pared to website and paper diary: pilot randomized controlled
trial. Journal of Medical Internet Research, 15(4), e32. doi:10.2196/
jmir.2283.
72. Chen, Z., Koh, P. W., Ritter, P. L., Lorig, K., Bantum, E. O. C., &
Saria, S. (2015). Dissecting an online intervention for cancer
survivors: four exploratory analyses of internet engagement
and its effects on health status and health behaviors. Health
Educ Behav, 42(1), 3245. doi:10.1177/1090198114550822.
73. Christensen, H., Griffiths, K. M., & Farrer, L. (2009). Adherencein
internet interventions for anxiety and depression. Journal of Med-
ical Internet Research, 11(2), e13. doi:10.2196/jmir.1194.
74. Crutzen, R., Cyr, D., & de Vries, N. K. (2012). The role of user
control in adherence to and knowledge gained from a website:
randomized comparison between a tunneled version and a
freedom-of-choice version. Journal of Medical Internet Research,
14(2), e45. doi:10.2196/jmir.1922.
75. Cussler, E. C., Teixeira, P. J., Going, S. B., Houtkooper, L. B.,
Metcalfe, L.L., Blew, R. M., etal. (2008). Maintenance of weight
loss in overweight middle-aged women through the internet.
Obesity, 16(5), 10521060. doi:10.1038/oby.2008.19.
76. Davies, C., Corry, K., Van Itallie, A., Vandelanotte, C., Caper-
chione, C., & Mummery, W. K. (2012). Prospective associations
between intervention components and website engagement in a
publicly available physical activity website: the case of 10,000
steps Australia. Journal of Medical Internet Research, 14(1), e4.
doi:10.2196/jmir.1792.
77. Dennison, L., Morrison, L., Lloyd, S., Phillips, D., Stuart, B.,
Williams,S., et al. (2014). Does brieftelephone support improve
engagement with a web-based weight management interven-
tion? Randomized controlled trial. Journal of Medical Internet Re-
search, 16(3), e95. doi:10.2196/jmir.3199.
78. Glasgow, R.E., Christiansen, S. M., Kurz, D., King, D. K., Woolley,
T., Faber, A. J., et al. (2011). Engagement in a diabetes self-
management website: usage patterns and generalizability of
program use. Journal of Medical Internet Research, 13(1), e9.
doi:10.2196/jmir.1391.
79. Manwaring, J. L., Bryson, S. W., Goldschmidt, A. B., Winzelberg,
A. J., Luce, K. H., Wilfley, D. E., & Taylor, C. B. (2008). Do
adherence variables predict outcome in an online program for
the prevention of eating disorders? J Consult Clin Psychol, 76(2),
341346. doi:10.1037/0022-006X.76.2.341.
80. Morrison, C., & Doherty, G. (2014). Analyzing engagement in a web-
based intervention platform through visualizing log-data. Journal of
Medical Internet Research, 16(11), e252. doi:10.2196/jmir.3575.
81. Murray, E., White, I. R., Varagunam, M., Godfrey, C., Khadjesari,
Z., & McCambridge, J. (2013). Attrition revisited: adherence and
retention in a web-based alcohol trial. Journal of Medical Internet
Research, 15(8), e162. doi:10.2196/jmir.2336.
82. Poirier, J., & Cobb, N. K. (2012). Social influence as a driver of
engagement in a web-based health intervention. Journal of Med-
ical Internet Research, 14(1), e36. doi:10.2196/jmir.1957.
83. Cugelman, B., Thelwall, M., & Dawes, P. (2011). Online interven-
tions for social marketing health behavior change campaigns: a
meta-analysis of psychological architectures and adherence fac-
tors. Journal of Medical Internet Research, 13(1), e17. doi:10.2196/
jmir.1367.
84. Henshaw, H., McCormack, A., & Ferguson, M. A. (2015). Intrinsic
and extrinsic motivation is associated with computer-based
auditorytraining uptake, engagement, and adherencefor people
with hearing loss. Front Psychol, 6,113. doi:10.3389/fpsyg.
2015.01067.
85. Hsu, C.-L., & Lu, H.-P. (2004). Why do people play on-line games? An
extendedTAMwith social influences and flow experience. Information
& Management, 41, 853868. doi:10.1016/j.im.2003.08.014.
86. McCabe, M. P., & Price, E. (2009). Attrition from an internet-
based psychological intervention for erectile dysfunction: who
is likely to drop out? Journal of Sex & Marital Therapy, 35(5), 391
401. doi:10.1080/00926230903065963.
87. Postel, M. G., de Haan, H. A., ter Huurne, E. D., van der Palen, J.,
Becker, E. S., & de Jong, C. A. J. (2011). Attrition in web-based
treatmentfor problem drinkers. Journal of Medical Internet Research,
13(4), e117. doi:10.2196/jmir.1811.
88. Johansson, O., Michel, T., Andersson, G., & Paxling, B. (2015).
Experiences of non-adherence to internet-delivered cognitive
behavior therapy: a qualitative study. Internet Interventions, 2,
137142. doi:10.1016/j.invent.2015.02.006.
89. Sainsbury, K., Mullan, B., & Sharpe, L. (2015). Dissemination of
an online theory-based intervention to improve gluten-free diet
adherence in coeliac disease: the relationship between
acceptability, effectiveness, and attrition. International Journal of
Behavioral Medicine, 22, 356364. doi:10.1007/s12529-014-
9416-4.
90. VanDeMark, N. R., Burrell, N. R., Lamendola, W. F., Hoich, C. A.,
Berg, N. P., & Medina, E. (2010). An exploratory studyof engage-
ment in a technology-supported substance abuse intervention.
Substance Abuse Treatment, Prevention, and Policy, 5(10), 114.
91. Al-Asadi, A. M., Klein, B., & Meyer, D. (2014). Pretreatment
attrition and formal withdrawal during treatment and their pre-
dictors: an exploratory study of the anxiety online data. Journalof
Medical Internet Research, 16(6), e152. doi:10.2196/jmir.2989.
92. Habibović, M., Cuijpers, P., Alings, M., van der Voort, P., Theuns,
D., Bouwels, L., et al. (2014). Attrition and adherence in a WEB-
based distress management program for implantable cardi-
overter defibrillator patients (WEBCARE): randomized controlled
trial. Journal of Medical Internet Research, 16(2), e52. doi:10.2196/
jmir.2809.
93. Hebert, E. A., Vincent, N., Lewycky, S., & Walsh, K. (2010).
Attrition and adherence in the online treatment of chronic in-
somnia. Behavioral Sleep Medicine, 8(3), 141150. doi:10.1080/
15402002.2010.487457.
94. Neve, M. J., Collins, C. E., & Morgan, P. J. (2010). Dropout, non-
usage attrition, and pretreatment predictors of nonusage attri-
tion in a commercial web-based weight loss program. Journal of
Medical Internet Research, 12(4), e69. doi:10.2196/jmir.1640.
95. Nicholas, J., Proudfoot, J., Parker, G., Gillis, I., Burckhardt, R.,
Manicavasagar, V., & Smith, M. (2010). The ins and outs of an
online bipolar education program: a study of program attrition.
Journal of Medical Internet Research, 12(5), e57. doi:10.2196/jmir.
1450.
96. Richardson, A., Graham, A. L., Cobb, N., Xiao, H., Mushro, A.,
Abrams, D., & Vallone, D. (2013). Engagement promotes absti-
nence in a web-based cessation intervention: cohort study.
Journal of Medical Internet Research, 15(1), e14. doi:10.2196/jmir.
2277.
97. Oinas-Kukkonen, H., & Harjumaa, M. (2009). Persuasive sys-
tems design: key issues, process model, and system features.
Commun Assoc Inf Syst, 24(28), 486501.
98. Hong, J.-C., Chiu, P.-Y., Shih, H.-F., & Lin, P.-S. (2012). Computer
self-efficacy, competitive anxiety and flow state: escaping from
firing online game. The Turkish Online Journal of Educational Tech-
nology, 11(3), 7076.
99. Meischke,H., Lozano, P., Zhou, C., Garrison, M. M., & Christakis,
D. (2011). Engagement in Bmy childsasthma^, an interactive
web-basedpediatric asthma management intervention.Int J Med
Inform, 80(11), 765774. doi:10.1016/j.ijmedinf.2011.08.002.
Engagement.
100. Boyle, E. A., Connolly, T. M., Hainey, T., & Boyle, J. M. (2012).
Engagement in digital entertainment games: a systematic re-
view. Comput Hum Behav, 28(3), 771780. doi:10.1016/j.chb.
2011.11.020.
101. Haines-Saah, R. J., Kelly, M. T., Oliffe, J. L., & Bottorff, J. L. (2015).
Picture Me Smokefree: a qualitative study using social media
and digital photography to engage young adults in tobacco
reduction and cessation. Journal of Medical Internet Research,
17(1), e27. doi:10.2196/jmir.4061.
102. Kim, Y. H., Kim,D. J., & Wachter, K. (2013). A studyof mobile user
engagement (MoEN): engagement motivations, perceived value,
satisfaction, and continued engagement intention. Decis Support
Syst, 56,361370. doi:10.1016/j.dss.2013.07.002.
103. Ludden, G. D., van Rompay, T. J., Kelders, S. M., & van Gemert-
Pijnen,J. E. (2015). How to increasereach and adherenceof web-
based interventions: a design research viewpoint. Journal of
Medical Internet Research, 17(7), e172. doi:10.2196/jmir.4201.
104. Parks, A. C. (2014). A case for the advancement of the design
and study of online positive psychological interventions. JPosit
Psychol, 9(6), 502508. doi:10.1080/17439760.2014.936969.
105. Horsch, C., Lancee, J., Beun, R. J., Neerincx, M. A., & Brink-
man, W.-P. (2015). Adherence to technology-mediated in-
somnia treatment: a meta-analysis, interviews, and focus
groups. JournalofMedicalInternetResearch,17(9), e214.
doi:10.2196/jmir.4115.
106. Funk, K. L., Stevens, V. J., Appel, L. J., Bauck, A., Brantley, P. J.,
Champagne, C. M., et al. (2010). Associations of internet website
use with weight change in a long-term weight loss maintenance
program. Journal of Medical Internet Research, 12(3), e29. doi:10.
2196/jmir.1504.
107. Graham, A. L., Cha, S., Cobb, N. K., Fang, Y., Niaura, R. S., &
Mushro, A. (2013). Impact of seasonality on recruitment, reten-
tion, adherence, and outcomes in a web-based smoking cessa-
tion intervention: randomized controlled trial. Journal of Medical
Internet Research, 15(11), e249. doi:10.2196/jmir.2880.
108. Peels,D. A., Bolman, C., Golsteijn, R. H. J., De Vries, H.,Mudde, A.
N., van Stralen, M. M., & Lechner, L. (2012). Differences in reach
and attrition between web-based and print-delivered tailored
interventions among adults over 50 years of age: clustered
SYSTEMATIC REVIEWS
TBMpage 12 of 14
randomized trial. Journal of Medical Internet Research, 14(6), e179.
doi:10.2196/jmir.2229.
109. Steinberg, D. M., Levine, E. L., Lane, I., Askew, S., Foley, P. B.,
Puleo, E., & Bennett, G. G. (2014). Adherence to self-monitoring
via interactive voice response technology in an eHealth interven-
tion targeting weight gain prevention among black women: ran-
domized controlled trial. Journal of Medical InternetResearch, 16(4),
e114. doi:10.2196/jmir.2996.
110. Strecher, V. J., McClure, J., Alexander, G., Chakraborty, B., Nair,
V., Konkel, J., et al. (2008). The role of engagement in a tailored
web-based smoking cessation program: randomized controlled
trial. Journal of Medical Internet Research, 10(5), e36. doi:10.2196/
jmir.1002.
111. Wanner, M., Martin-Diener,E., Bauer, G., Braun-Fahrländer, C., &
Martin, B. W. (2010). Comparison of trial participants and open
access users of a web-based physical activity intervention re-
garding adherence, attrition, and repeated participation. Journal
of Medical Internet Research, 12(1), e3. doi:10.2196/jmir.1361.
112. Jahangiry, L., Shojaeizadeh, D., Montazeri, A., & Najafi, M.
(2014). Adherence and attrition in a web-based lifestyle inter-
vention for people with metabolic syndrome. Iranian Journal of
Public Health, 43(9), 12481258.
113. Kuijpers, W., Groen, W. G., Aaronson, N. K., & van Harten, W. H.
(2013). A systematic review of web-based interventions for pa-
tient empowerment and physical activity in chronic diseases:
relevance for cancer survivors. Journal of Medical Internet Research,
15(2), e37. doi:10.2196/jmir.2281.
114. Mahmassani, H. S., Chen, R. B., Huang, Y., Williams, D., &
Contractor, N. (2010). Time to play? Activity engagement in
multiplayer online role-playing games. Transportation Research
Record: Journal of the Transportation Research Board, 2157,129
137. doi:10.3141/2157-16.
115. Ferguson, M. A., & Henshaw, H. (2015). Computer and internet
interventions to optimize listening and learning for people with
hearing loss: accessibility, use, and adherence. Am J Audiol, 24,
338343. doi:10.1044/2015.
116. Weston A, Morrison L, Yardley L, Van Kleek M, Weal M. Measure-
ments of engagement in mobile behavioural interventions? In
Digital Health. 2015:18.
117. Donovan,E., Mahapatra, P.D., Green, T. C., Chiauzzi, E., Mchugh,
K., Hemm, A., et al. (2015). Efficacy of an online intervention to
reduce alcohol-related risks amongcommunity college students.
Addiction Research & Theory, 23(5), 437447. doi:10.3109/
16066359.2015.1043625.
118. Khadjesari, Z., Murray, E., Kalaitzaki, E., White, I. R., McCam-
bridge, J., Thompson, S. G., et al. (2011). Impact and costs of
incentives to reduce attrition in online trials: two randomized
controlled trials. Journal of Medical Internet Research, 13(1), e26.
doi:10.2196/jmir.1523.
119. An, L. C., Perry, C. L., Lein, E. B., Klatt, C., Farley, D. M., Bliss, R. L.,
et al. (2006). Strategies for increasing adherence to an online
smoking cessation intervention for college students. Nicotine &
Tobacco Research, 8(December), S7S12. doi:10.1080/
14622200601039881.
120. Brouwer, W., Kroeze, W., Crutzen, R., de Nooijer, J., de Vries, N.
K., Brug, J., & Oenema, A. (2011). Which intervention character-
istics are related to more exposure to internet-delivered healthy
lifestyle promotion interventions? A systematic review. Journal of
Medical Internet Research, 13(1), e2. doi:10.2196/jmir.1639.
121. Cairns, P., Cox, A. L., Day, M., Martin, H., & Perryman, T. (2013).
Who but not where: the effect of social play on immersion in
digital games. Int J Hum Comput Stud, 71, 10691077. doi:10.
1016/j.ijhcs.2013.08.015.
122. Morris,R. R., Schueller, S. M.,& Picard, R. W. (2015).Efficacy of a
web-based, crowdsourced peer-to-peer cognitive reappraisal
platform for depression: randomized controlled trial. Journal of
Medical Internet Research, 17(3), e72. doi:10.2196/jmir.4167.
123. Crutzen, R., Cyr, D., Larios, H., Ruiter, R. A. C., & De Vries, N. K.
(2013). Social presence and use of internet-delivered interven-
tions: a multi-method approach. PLoS One, 8(2), e57067. doi:10.
1371/journal.pone.0057067.
124. Ben-Zeev, D., Kaiser, S. M.,& Krzos, I. (2014). Remote Bhovering^
with individuals with psychotic disorders and substance use:
feasibility, engagement, and therapeutic alliance with a text-
messaging mobile interventionist. Journal of Dual Diagnosis,
10(4), 197203. doi:10.1080/15504263.2014.962336.Remote.
125. Miller, A. S., Cafazzo, J. A., & Seto, E. (2014). A game plan:
gamification design principles in mHealth applications for chron-
ic disease management. Health Informatics Journal,110. doi:10.
1177/1460458214537511.
126. Brigham, T. J. (2015). An introduction to gamification: adding
game elementsfor engagement. Medical Reference Services Quarter-
ly, 34(4), 471480. doi:10.1080/02763869.2015.1082385.
127. Richardson, C. R., Buis, L. R., Janney, A. W., Goodrich, D. E., Sen,
A., Hess, M. L., et al. (2010). An online community improves
adherence in an internet-mediated walking program. Part 1:
results of a randomized controlled trial. Journal of Medical Internet
Research, 12(4), e71. doi:10.2196/jmir.1338.
128. Leslie, E., Marshall, A. L., Owen, N., & Bauman, A. (2005).
Engagement and retention of participants in a physical activity
website. Preventive, 40,5459. doi:10.1016/j.ypmed.2004.05.
002.
129. Irvine,A. B., Russell, H., Manocchia, M., Mino, D. E.,Cox Glassen,
T., Morgan, R., et al. (2015). Mobile-web app to self-manage low
back pain: randomized controlled trial. Journal of Medical Internet
Research, 17(1), e1. doi:10.2196/jmir.313 0.
130. Lin, H., & Wu, X. (2014). Intervention strategies for improving
patient adherence to follow-up in the era of mobile information
technology: a systematic review and meta-analysis. PLoS One,
9(8), e104266. doi:10.1371/journal.pone.0104266.
131. Kok, G., Bockting, C., Burger, H., Smit, F., & Riper, H. (2014).
Mobile cognitive therapy: adherence and acceptability of an
online intervention in remitted recurrently depressed patients.
Internet Interventions, 1,6573. doi:10.1016/j.invent.2014.05.
002.
132. van den Berg, M. H., Ronday, H. K., Peeters, A. J., Voogt-van der
Harst, E. M., Munneke, M., Breedveld, F. C., & Vliet Vlieland, T. P.
M. (2007). Engagement and satisfaction with an internet-based
physical activity intervention in patients with rheumatoid arthri-
tis. Rheumatology, 46(3), 545552. doi:10.1093/rheumatology/
kel341.
133. Stark, S., Snetselaar, L., Piraino, B., Stone, A., Kim, S., Hall, B., &
Burke, L. E. (2011). PDA self-monitoring adherence rates in two
dialysis dietary intervention pilot studies: BalanceWise-HD and
BalanceWise-PD. JRenNutr,21(6), 492498. doi:10.1053/j.jrn.
2010.10.026.PDA.
134. Mohr, D. C.,Duffecy, J., Ho, J., Kwasny, M., Cai, X., Burns, M. N., &
Begale, M. (2013). A randomized controlled trial evaluating a
manualized TeleCoaching protocol for improving adherence to a
web-based intervention for the treatment of depression. PLoS
One, 8(8), e70086. doi:10.1371/journal.pone.0070086.
135. Klein, M., Mogles, N., & Wissen, A. V. (2014). Intelligent mobile
support for therapy adherence and behavior change. J Biomed
Inform, 51,137151. doi:10.1016/j.jbi.2014.05.005.
136. McCambridge, J., Kalaitzaki, E., White, I. R., Khadjesari, Z., Mur-
ray, E., Linke, S., et al. (2011). Impact of length or relevance of
questionnaires on attrition in online trials: randomized con-
trolled trial. Journal of Medical Internet Research, 13(4), e96.
doi:10.2196/jmir.1733.
137. Helander, E., Kaipainen, K., Korhonen, I., & Wansink, B. (2014).
Factors related to sustained use of a free mobile app for dietary
self-monitoring with photography and peer feedback: retrospec-
tive cohort study. Journal of Medical Internet Research, 16(4), e109.
doi:10.2196/jmir.3084.
138. Whiteside, U., Lungu, A., Richards, J., Simon, G. E., Clingan, S.,
Siler, J., etal. (2014). Designing messaging to engage patients in
an online suicide prevention intervention: survey results from
patients with current suicidal ideation. Journal of Medical Internet
Research, 16(2), e42. doi:10.2196/jmir.3173.
139. Jennings, M. (2000). Theory and models for creating engaging
and immersive e-commerce websites. In Proceedings of the 2000
ACM SIGCPR Conference on Computer Personnel Research (pp. 77
85). New York: ACM.
140. Park, N.,Min, K., Jin, S. A., & Kang, S.(2010). Effects of pre-game
stories on feelings of presence and evaluation of computer
games. Int J Hum Comput Stud, 68, 822833. doi:10.1016/j.
ijhcs.2010.07.002.
141. Hwang, M.-Y., Hong, J.-C., Hao, Y.-W., & Jong, J.-T. (2011). Elders
usability, dependability, and flow experiences on embodied
interactive video games. Educ Gerontol, 37(8), 715731. doi:10.
1080/03601271003723636.
142. Chapman P, Selvarajah S, Webster J. Engagement in multimedia
training systems. In Proceedings of the 32nd Hawaii International
Conference on System Sciences 1999; 0: 19. Washington, DC: IEEE.
doi:10.1109/HICSS.1999.772808.
143. Liu, S., Liao, H., & Pratt, J. A. (2009). Impact of media richness
and flow on e-learning technology acceptance. Comput Educ, 52,
599607. doi:10.1016/j.compedu.2008.11.002.
144. Miller, A. S., Cafazzo, J. A., & Seto, E. (2014). A game plan:
gamification design principles in mHealth applications for chron-
ic disease management. Health Informatics Journal. doi:10.1177/
1460458214537511.
145. Lieberman, D. Z. (2006). Effects of a personifiedguide on adher-
ence to an online program for alcohol abusers. Cyberpsychology &
Behavior, 9(5), 603607.
146. Bellg,A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory,
M., et al. (2004). Enhancing treatment fidelity in health behavior
change studies: best practices and recommendations from the
NIH Behavior Change Consortium. Health Psychol, 23(5), 443
451. doi:10.1037/0278-6133.23.5.443.
147. Borrelli, B. (2011). The assessment, monitoring, and enhance-
ment of treatment fidelity in public health clinical trials. JPublic
SYSTEMATIC REVIEWS
TBM page 13 of 14
Health Dent, 71,S52S63. doi:10.1111/j.1752-7325.2011.
00233.x.
148. Ubhi, H. K.,Michie, S., Kotz, D., Wong, W. C., & West, R. (2015).A
mobile app to aid smoking cessation: preliminary evaluation of
SmokeFree28. Journal of Medical Internet Research, 17(1), e17.
doi:10.2196/jmir.3479.
149. Chinn, P. L., & Kramer, M. K. (1991). Theory and nursing: a system-
atic approach. St. Louis: Mosby-Year Book.
150. Fiannaca A, La Rosa M, Rizzo R, Urso A, Gaglio S. An ontology
design methodology for Knowledge-Based systems with applica-
tion to bioinformatics. In Computational Intelligence in Bioinfor-
matics and Computational Biology (CIBCB),2012 I.E. Symposium.
2012:8591.
151. Weber, R. (2012). Evaluating and developing theories in the
information systems discipline. J Assoc Inf Syst, 13(1), 130.
152. OBrien, H. L., & Toms, E. G. (2010). The development and
evaluation of a survey to measure user engagement. Journal of
the AmericanSociety for Information Science& Technology, 61(1), 50
69. doi:10.1002/asi.21229.
153. Stone, A. A., & Shiffman, S. (1994). Ecological momentary as-
sessment (EMA) in behavorial medicine. Ann Behav Med, 16(3),
199202.
154. Haukkala, A., Uutela, A., Vartiainen, E., Mcalister, A., & Knekt, P.
(2000). Depression and smoking cessation: the role of motiva-
tion and self-efficacy. Addict Behav, 25. doi:10.1016/S0306-
4603(98)00125-7.
155. Linde, J. A., Jeffery, R. W., Levy, R. L., Sherwood, N. E., Utter, J.,
Pronk, N. P., & Boyle, R. G. (2004). Binge eating disorder, weight
control self-efficacy, and depression in overweight men and
women. Int J Obes, 28(3), 418425. doi:10.1038/sj.ijo.0802570.
SYSTEMATIC REVIEWS
TBMpage 14 of 14

Supplementary resource (1)

... User engagement in digital behavior change interventions has predicted improved treatment outcomes across a wide variety of domains, including mental health, physical activity, dietary change, weight loss, alcohol use, and smoking cessation [1][2][3][4][5][6][7][8][9]. A central challenge in creating effective digital health behavior change interventions is that a large proportion of users disengage early from these interventions, thereby contributing to low treatment success rates [2,10,11]. ...
... User engagement in digital behavior change interventions has predicted improved treatment outcomes across a wide variety of domains, including mental health, physical activity, dietary change, weight loss, alcohol use, and smoking cessation [1][2][3][4][5][6][7][8][9]. A central challenge in creating effective digital health behavior change interventions is that a large proportion of users disengage early from these interventions, thereby contributing to low treatment success rates [2,10,11]. Given the importance of user engagement, designing strategies to increase engagement has been a priority of digital behavioral health interventions [2,7,8,12,13]. ...
... A central challenge in creating effective digital health behavior change interventions is that a large proportion of users disengage early from these interventions, thereby contributing to low treatment success rates [2,10,11]. Given the importance of user engagement, designing strategies to increase engagement has been a priority of digital behavioral health interventions [2,7,8,12,13]. ...
Article
BACKGROUND: Little is known about how individuals engage over time with smartphone app interventions and whether this engagement predicts health outcomes. OBJECTIVE: In the context of a randomized trial comparing 2 smartphone apps for smoking cessation, this study aimed to determine distinct groups of smartphone app log-in trajectories over a 6-month period, their association with smoking cessation outcomes at 12 months, and baseline user characteristics that predict data-driven trajectory group membership. METHODS: Functional clustering of 182 consecutive days of smoothed log-in data from both arms of a large (N=2415) randomized trial of 2 smartphone apps for smoking cessation (iCanQuit and QuitGuide) was used to identify distinct trajectory groups. Logistic regression was used to determine the association of group membership with the primary outcome of 30-day point prevalence of smoking abstinence at 12 months. Finally, the baseline characteristics associated with group membership were examined using logistic and multinomial logistic regression. The analyses were conducted separately for each app. RESULTS: For iCanQuit, participants were clustered into 3 groups: "1-week users" (610/1069, 57.06%), "4-week users" (303/1069, 28.34%), and "26-week users" (156/1069, 14.59%). For smoking cessation rates at the 12-month follow-up, compared with 1-week users, 4-week users had 50% higher odds of cessation (30% vs 23%; odds ratio [OR] 1.50, 95% CI 1.05-2.14; P=.03), whereas 26-week users had 397% higher odds (56% vs 23%; OR 4.97, 95% CI 3.31-7.52; P<.001). For QuitGuide, participants were clustered into 2 groups: "1-week users" (695/1064, 65.32%) and "3-week users" (369/1064, 34.68%). The difference in the odds of being abstinent at 12 months for 3-week users versus 1-week users was minimal (23% vs 21%; OR 1.16, 95% CI 0.84-1.62; P=.37). Different baseline characteristics predicted the trajectory group membership for each app. CONCLUSIONS: Patterns of 1-, 3-, and 4-week smartphone app use for smoking cessation may be common in how people engage in digital health interventions. There were significantly higher odds of quitting smoking among 4-week users and especially among 26-week users of the iCanQuit app. To improve study outcomes, strategies for detecting users who disengage early from these interventions (1-week users) and proactively offering them a more intensive intervention could be fruitful.
... This study also contributes to the mHealth literature by outlining an mHealth intervention development method that may be useful in promoting app engagement. Engagement with apps can defined as "(1) the extent (eg, amount, frequency, duration, and depth) of use and (2) a subjective experience characterized by attention, interest, and affect" [60]. Engagement with mHealth programs is typically poor, which leads to insufficient behavior change (eg, [61]). ...
... Engagement with mHealth programs is typically poor, which leads to insufficient behavior change (eg, [61]). Factors that increase engagement in mHealth interventions include using BCTs (eg, self-monitoring and action planning) and providing health care practitioner support [60,61]. By conducting a behavioral analysis and working with a technology partner who endorses social support from a health coach, the SCI Step Together program is more likely to support participant engagement. ...
... By conducting a behavioral analysis and working with a technology partner who endorses social support from a health coach, the SCI Step Together program is more likely to support participant engagement. Other factors that influence app engagement include "safety netting" (ie, having the ability to re-engage with the app after disengaging) and tailored content [60,61]. Using IKT allowed us to work with end users to create tailored physical activity content that could be individualized in the app through the health coach. ...
Article
Background Interventions to support physical activity participation among individuals with spinal cord injury (SCI) are required given this population’s low levels of physical activity and extensive barriers to quality physical activity experiences. Objective This study aimed to develop a mobile health intervention, called SCI Step Together, to improve the quantity and quality of physical activity among individuals with SCI who walk. Methods Our overarching methodological framework was the Person-Based approach. This included the following 4 steps: conduct primary and secondary research (step 1); design intervention objectives and features (step 2a); conduct behavioral analysis and theory (step 2b); create a logic model (step 3); and complete the SCI Step Together program content and integrated knowledge translation (IKT; step 4), which occurred throughout development. The partnership approach was informed by the SCI IKT Guiding Principles. Three end users pilot-tested the app and participated in the interviews. Results Step 1 identified issues to be addressed when designing intervention objectives and features (step 2a) and features were mapped onto the Behavior Change Wheel (step 2b) to determine the behavior change techniques (eg, action planning) to be included in the app. The logic model linked the mechanisms of action to self-determination theory (steps 2/3). Interviews with end users generated recommendations for the technology (eg, comparing physical activity levels with guidelines), trial (eg, emailing participants’ worksheets), and intervention content (eg, removing graded tasks; step 4). Conclusions Using the SCI IKT Guiding Principles to guide partner engagement and involvement ensured that design partners had shared decision-making power in intervention development. Equal decision-making power maximizes the meaningfulness of the app for end users. Future research will include testing the acceptability, feasibility, and engagement of the program. Partners will be involved throughout the research process. Trial Registration ClinicalTrials.gov: NCT05063617; https://clinicaltrials.gov/ct2/show/NCT05063617
... 18 Poor engagement is commonly observed with smartphone apps, contributing to their insufficiency for sustaining behavior change, and evidence is lacking regarding the main factors contributing to this problem. 19,20 In addition, studies assessing the relationship between user engagement and dietary self-care behaviors for the prevention of cardiovascular disease are limited. 21 Furthermore, few studies identify distinct engagement trajectories, which may be helpful to characterize and predict engagement patterns and make digital behavior change interventions more effective. ...
... Others have identified education as a significant predictor of engagement. 19,50 In our study, this may reflect skills and confidence with using Nutritionix and social norms related to the perceived value of dietary tracking. 25 While it is important to uncover variations in engagement by sociodemographic characteristics, what these differences highlight is the importance of assessing the needs and skills of the target population to ensure a high-quality digital intervention that is accessible to all. ...
Article
Full-text available
Background The DASH dietary pattern is a proven way to manage hypertension, but adherence remains low. Dietary tracking apps offer a highly disseminable way to self-monitor intake on the pathway to reaching dietary goals but require consistent engagement to support behavior change. Few studies use longitudinal dietary self-monitoring data to assess trajectories and predictors of engagement. We used dietary self-monitoring data from participants in DASH Cloud (n=59), a feasibility trial to improve diet quality among women with hypertension, to identify trajectories of engagement and explore associations between participant characteristics. Methods We used latent class growth modeling to identify trajectories of engagement with a publicly available diet tracking app and used bivariate and regression analyses to assess associations of classifications of engagement with participant characteristics. Results We identified two latent classes of engagement, consistent engagers and disengagers. Consistent engagers were more likely to be older, more educated, and married or living with a partner. Although consistent engagers exhibited slightly greater changes in DASH score, the difference was not significant. Conclusion This study highlights an important yet underutilized methodological approach for uncovering dietary self-monitoring engagement patterns. Understanding how certain individuals engage with digital technologies is an important step toward designing cost-effective behavior change interventions. Trial registration NCT03215472
... Further research on the virtual care needs of patients with HMs is warranted; surveys and validated questionnaires explicitly designed for patients with HMs are needed to gain knowledge of the unmet needs of these patients. Understanding patient preferences through a continuous user-centered design is essential for the success of mHealth for patients with HMs [50,55]. The differences found between features included in apps and the preferences of patients with HMs highlight the apparent necessity of their participation in the design of health-related apps. ...
Article
Background Hematological malignancies (HMs) are a heterogeneous group of cancers representing a significant cause of morbidity and mortality. The chronification of HMs and the increasing use of smartphones may lead patients to seek their current unmet needs through mobile health apps. Objective The goal of this review was to identify and assess the quality of smartphone apps aimed at patients diagnosed with HMs. Methods A systematic search of apps that were aimed at patients diagnosed with HMs, accessed from a Spain IP address, and were available on the iOS (App Store) and Android (Google Play) platforms was conducted in November 2021. The search terms used were “hematology,” “blood cancer,” “leukemia,” “lymphoma,” and “myeloma” apps in English, Spanish, or both languages. The identified apps were downloaded and analyzed independently by 2 reviewers. Information about general app characteristics was collected. The Mobile Application Rating Scale (MARS) was used to assess quality. The resulting parameter of the analyses, the mean score of the apps, was compared by Student t test. Results Overall, 18 apps were identified; 7 were available on Android, 5 were available on iOS, and 6 were available on both platforms. All included apps were free; 3 were published in 2021, and among the apps published before 2021, only 6 were updated in 2021. Most (16/18, 89%) of the apps were aimed at patients with leukemia or lymphoma (16). The primary purposes of the apps were to provide general information about the condition (16/18, 89%) and monitor symptoms and clinical parameters (11/18, 61%). Health care professionals contributed to the development of 50% (9/18) of apps; 6 were owned and supported by scientific societies, and 3 were developed with the participation of health care professionals. The mean MARS score for the overall quality of the apps was 3.1 (SD 1.0). The engagement and aesthetics subscales were the lowest rated subscales, with only 44% (8/18) and 67% (12/18), respectively, of the apps obtaining acceptable scores. None of the included apps proved clinical efficacy through clinical trials in patients with HMs. Statistically significant differences were found in the MARS scores between operating systems (+1.0, P=.003) in favor of iOS apps. The participation of health care professionals in the development of the apps did not have a statistically significant impact on the MARS scores. Conclusions This systematic search and evaluation identified few acceptable quality mobile apps for patients with HMs. Current and future apps for patients with HMs should provide evidence-based valuable information, improve user engagement, incorporate functions according to patient preferences, and generate evidence regarding the efficacy of app use by patients with HMs.
... Perceived effectiveness refers to the perceived benefits that provide the use of technology and whether it appears likely to achieve its purpose [76]. Engagement refers to the use and intention to continue using a technology [78]. ...
Article
Full-text available
The use of mobile health apps to improve diet and nutrition behaviors has increased in recent years. Several studies have described the benefits and advantages of this technology as a complement to interventions for improving nutrition behaviors and nutrition‐related health outcomes, including obesity indices and clinical parameters. Few of these works have developed clinical mobile health apps for children, and although parents play a critical role in children’s nutrition behaviors, work targeting parents is scarce. The work presented in this paper describes the development of the PersuHabit app, a stand-alone mobile health app targeting parents to promote the intake of fruits and vegetables (FVs) and reduce the intake of ultra-processed foods (UPF) in children aged 6 to 10 years. The paper also presents the execution of an exploratory pilot study to assess the feasibility, acceptability, and preliminary effects of the PersuHabit app. The results are presented and discussed, and actions for further improvement of the PersuHabit app are identified.
... may be worth exploring in an attempt to help practitioners scale their service delivery in a time and cost-effective manner. During these processes it is recommended that that the acceptability of any novel applications, as well as athletes' engagement with these technologies, is explored (Perski et al., 2016;Perski and Short, 2021). ...