- Access to this full-text is provided by De Gruyter.
Download available
Content available from it - Information Technology
This content is subject to copyright. Terms and conditions apply.
Digital health interventions (DHIs) have the po-
tential to help the growing number of chronic disease pa-
tients better manage their everyday lives. However, guide-
lines for the systematic development of DHIs are still
scarce. The current work has, therefore, the objective to
propose a framework for the design and evaluation of DHIs
(DEDHI). The DEDHI framework is meant to support both
researchers and practitioners alike from early conceptual
DHI models to large-scale implementations of DHIs in the
healthcare market.
barriers, criteria, digital health intervention,
evaluation, life cycle, recommendations
CCS →Applied computing →Life and medical
sciences →Health care information systems
Over the last decades, the prevalence of chronic health
problems, i. e. diseases, conditions, and syndromes that
are continuing or occurring repeatedly for a long time, is
steadily increasing. Chronic health problems include, for
example, cardiovascular diseases, diabetes, chronic respi-
ratory diseases (e. g., COPD or asthma), arthritis or certain
types of cancer (e. g., multiple myeloma) [10, 46, 53]. These
health problems lead not only to a substantial decrease in
the quality of life of those being aected [33, 41, 68] or loss
in productivity [64] but represent also the most important
economic challenge in developed countries with up to 86
percent of all healthcare expenditures [12, 46, 53].
In addition to current approaches to address this im-
portant problem, for example, through national chronic
disease strategies and policies [89], the use of information
technology to either monitor health conditions and behav-
ior or to deliver health interventions is another promis-
ing approach to support the growing number of chronic
patients in their everyday lives [1, 53]. In this article, we
use the term digital health intervention (DHI), describing
the action of intervening [66] with “tools and services that
use information and communication technologies (ICTs)
to improve prevention, diagnosis, treatment, monitoring
and management of health and lifestyle” [26], which is
closely related to the notion of mHealth, telemedicine,
telecare and health IT [28].
In light of the mature history of evidence-based
medicine with clear guidelines on how to develop and as-
sess the eectiveness of biomedical or behavioral health
interventions [70, 82], up till now, guidelines for the sys-
tematic development and assessment of DHIs are still
scarce and corresponding research has just started. For ex-
ample, rst evaluation criteria have been proposed during
the last decade [8, 9, 18, 27, 60, 80, 83]. However, this pre-
liminary work lacks guidance to which degree and when
to apply these criteria along the life cycle of DHIs [63, 78].
Particularly, it is essential to consider appropriate evalua-
tion criteria not only during the conceptual and prototype
phases of a DHI, but also with respect to long-term imple-
mentations in the health care market, so that a sustain-
able, eective and ecient use of DHIs can be achieved.
The evaluation results would also be the foundation for
trust-building certications similar to energy eciency la-
bels of consumer products, which can be used by patients
and health professionals alike to nd the “right” DHIs.
Moreover, barriers for implementation and scaling-up of
DHIs remain [65] that intervention authors must be aware
of and that need to be addressed during and after the de-
velopment process. A successful DHI conclusively needs
to consider both, the selection of suitable evaluation crite-
ria and the overcoming of implementation barriers. There-
fore, a match-making is deemed useful to assess which
evaluation criteria need to be considered and which im-
plementation barriers need to be addressed at which par-
ticular phase of a DHI life cycle.
The current work has therefore the objective to pro-
pose a framework for the iterative Design and Evaluation
of DHIs (DEDHI) that describes a typical life cycle of a DHI
and recommends relevant evaluation criteria and imple-
mentation barriers to be considered for each phase of this
life cycle.
The DEDHI framework is meant to support both re-
searchers and practitioners alike during the design and
evaluation of various instantiations of DHIs, i. e., from
conceptual models to large-scale implementations in the
healthcare area. The scientic contribution lies in the
alignment of research streams from dierent elds at the
intersection of behavioral medicine (e. g., behavioral in-
terventions), medical informatics (e. g., medical applica-
tions) and information systems research (e.g., barriers of
health information systems, including aspects of technol-
ogy acceptance).
The remainder of this article is structured as follows.
Design and evaluation frameworks for health interven-
tions and DHI life cycle models are presented in the next
section which build the foundation of the proposed DEDHI
framework. For this purpose, an extended version of the
multiphase-optimization strategy (MOST) [19, 20, 21] is
used as the guiding life cycle model. Then, a systematic
literature review is described and a consolidated list of
evaluation criteria for DHIs are presented. Afterwards and
based on a previous literature review [65], a consolidated
list of implementation barriers for DHIs are outlined. In
the following main results section, the consolidated eval-
uation criteria and implementation barriers for DHIs are
both mapped to the DEDHI framework. This mapping is
conducted in a deductive manner by applying qualitative
content analysis [55]. Finally, the resulting DEDHI frame-
work is discussed with recommendations for research and
practice, and limitations. A summary and suggestions for
future work conclude this article.
Various design and evaluation frameworks for health in-
terventions have been proposed in the past. Examples of
these frameworks are listed in Table 1. They range from
guidelines for the development of public health interven-
tions [87] and policies [24] at the population-level to be-
havioral health interventions [56] and DHIs [59] at the
individual-level. A common shortcoming of these frame-
works, however, lies in the lack of guidance with respect to
evaluation criteria and implementation barriers along the
dierent phases of a typical DHI life cycle. That is, appro-
priate guidance is missing from the conceptual model of a
DHI to a product-grade DHI that is maintained in the long-
term. In particular, none of these frameworks oers guid-
ance on technology-related aspects (e. g. maturity, scala-
bility or security) and there are only a few frameworks that
consider the implementation phase explicitly [16, 22].
To address these shortcomings, ndings from DHI life-
cycle models [13, 39, 47, 77] can be used. These mod-
els describe the phases that systems undergo while they
evolve from a prototypical development to an operational
product [84]. For example, Broens etal. [13] proposed a
four-layered life-cycle model. It distinguishes between the
phases of prototypes, small-scale pilots, large-scale pilots,
operational product and links specic determinants of
successful DHI implementations to each of these phases.
The generic Technology Readiness Level model also fol-
lows this structure but renes the initialization and pro-
totype phases in a more granular way [54].
Against this background and in order to account for
all relevant phases of DHI development and implementa-
tion, we propose an extended version of MOST [19, 20, 21]
as the guiding life cycle model for DEDHI. It was selected,
because (a) it describes the development of DHIs in a rig-
orous and iterative way with several design, optimization
and evaluation steps and clearly dened optimization cri-
teria, (b) it explicitly considers a novel class of personal-
ized and promising health interventions, i.e., just-in-time
adaptive interventions [61, 62] and corresponding assess-
ment methods such as micro-randomized trials [48] that
heavily rely on the use of technology and, nally, (c) be-
cause it also focuses on behavioral health interventions
at the individual-level which is relevant for chronic health
problems [46, 53]. Due to the fact that MOST does not con-
sider a phase after a DHI has been successfully evaluated
in a randomized controlled trial, a corresponding imple-
mentation phase is added from both related design and
evaluation frameworks from Table 1 [15, 16, 22] and a DHI
life cycle model [13]. Moreover, details on recommended
maturity levels of DHI technology are also incorporated
into this extended version of MOST from corresponding
DHI life cycle models [13, 54].
The proposed DEDHI framework, which is based upon
this extended version of MOST, is shown in Table 6. This ta-
ble also includes the consolidated evaluation criteria and
implementation barriers for DHIs which are described in
more detail in the following two sections.
A systematic literature review was conducted to identify
evaluation criteria for DHIs. A recently published system-
atic review of quality criteria for mobile health applica-
tions [63] in combination with an explorative search in the
PubMed and Google Scholar databases were used to iden-
tify appropriate search terms that revealed a signicant
amount of relevant search results.
The nal set of search terms is listed in Table 2 and
was applied as follows: (ID1 and ID2 and ID3 and ID4 in
Title) and (ID1 and ID2 in Abstract) (note that ID refers to
the search term ID listed in Table 2).
The goal of the search strategy was to update and com-
plement prior ndings [63] due to the broader focus of
the current work on DHIs which includes not only mobile
health interventions but also web-based interventions and
hybrid interventions in which also guidance by human
health professionals are foreseen [51]. The resulting search
strategy therefore consisted of three approaches. First, a
backward search was conducted with relevant work al-
ready identied by Nouri et al. [63] but with the broader
focus on DHIs. Here, relevant articles were screened back
to the year 2000, which can be determined as the start
of systematic research on DHIs [3, 4]. Second, the work
of Nouri et al. [63] was updated with the broader DHI fo-
cus and relevant work from December 2016 till May 2019.
Third, the search strategy of Nouri et al. [63] was extended
to socio-technical databases and journals, i. e., ACM Digi-
tal Library, IEEE Explore, and A-ranked and B-ranked dig-
ital health journals as listed in [75]. An overview of the
search strategy is outlined in Table 3.
A search result was included if the work was origi-
nal, peer-reviewed, written in English, and described a
tool with evaluation criteria for DHIs. Thus, systematic re-
views of evaluation criteria were excluded but relevant
work from these reviews was screened when published be-
tween January 2000 and May 2019.
The inclusion of relevant work was initially carried
out by two authors of this article on the basis of title and
abstract. In the event of uncertainty as to whether a par-
ticular work fullled the inclusion criteria, the entire text
was read and, if necessary, a third co-author was con-
sulted. The evaluation criteria with a corresponding def-
inition were then extracted from the resulting list of in-
cluded work. All criteria with corresponding denitions (if
available) were then reviewed independently by two co-
authors and summarized into inductive categories accord-
ing to qualitative content analysis [55]. In case of uncer-
tainty, the two co-authors consulted each other and also
included a third co-author to nd a consensus.
The systematic search led initially to 2616 journal arti-
cles and conference papers which were then screened step
by step as outlined in Figure 1.
Overall, 331 evaluation criteria were then extracted
from the resulting 36 records and consolidated into 13 cate-
gories. These categories are listed in Table 4 and accompa-
nied by a description, references for further readings and
the number of corresponding evaluation criteria.
=
An overview of all selected articles, evaluation crite-
ria and mapping of these criteria to the categories includ-
ing examples is provided in [50]. The results of the con-
solidated categories show that ease of use is by far the
most dominant category, with 87 evaluation criteria. By
contrast, evaluation criteria related to ethical and safety
aspects of a DHI are so far quite neglected by the scien-
tic community. Moreover, it can be observed that one fun-
damental aspect of evidence-based medicine and the pri-
mary objective of design and evaluation frameworks as
outlined in Table 1, i.e., to assess the degree to which an
intervention is eective, does not take over a prominent
position with the eighth rank in Table 4. Finally, it can be
noticed that both subjective evaluation criteria (e. g., per-
ceived benet of a DHI) and criteria measured objectively
(e. g., adherence to a DHI) are listed among the resulting
categories.
A list of implementation barriers of DHIs was already
identied in prior work by means of a systematic litera-
ture review of reviews [65]. For the purpose of the current
work, the 98 identied implementation barriers were sum-
marized into inductive categories according to qualitative
content analysis [55]. Out of the 98 barriers, 106 assign-
ments to categories could be made. This higher number is
due to the fact, that some barriers are related to more than
one category. An overview of the resulting categories of im-
plementation barriers, their descriptions and numbers are
shown in Table 5.
The mapping of the evaluation criteria and implementa-
tion barriers for DHIs along the life cycle phases of the
proposed DEDHI framework was conducted by means of
a qualitative content analysis [55]. The analysis was done
by at least two scientists independently, whereby inconsis-
tencies were resolved through discussion until consensus
was reached. The resulting overview of the DEDHI frame-
work, including the mapping of evaluation criteria and im-
plementation barriers, is listed in Table 6.
For each phase of the DEDHI framework, the overall
goal and corresponding design and evaluation tasks are
outlined. These goals and tasks are adapted to the concept
=
of DHIs from MOST [19, 20, 21] for the Phases 1, 2 and 3
and from related work on intervention design and life cy-
cle models [15, 16, 22] for Phase 4 as outlined in Section 2.
In addition, a brief description of the technical maturity
of the DHI is provided to help intervention authors better
understand the technical perspective. Moreover, relevant
evaluation criteria and implementation barriers are pro-
vided for each phase of the DEDHI framework that are sug-
gested to be addressed by intervention authors in order to
create evidence-based DHIs that can be successfully im-
plemented in the health care market.
While almost all criteria and barriers are only related
to a single phase, some are related to two or all phases.
For example, the two barrier categories funding and cost
are related to all the phases as they represent start-up
as well as maintenance cost and funding. Also, some in-
dividual characteristics (e.g., lack of trust in colleagues
[43, 71], lack of trust in politics [71], sticking to old fash-
ioned modalities of care [43]) and negative associations
of healthcare providers relate to more than one phase.
First, they need to be considered within user-centered de-
sign processes in the preparation phase. Second, they can
be addressed during the implementation phase by means
of advertisement and awareness campaigns. Furthermore,
usability also relates to more than one phase. However,
review existing justicatory
knowledge
develop a
conceptual model
conduct a feasibility and
acceptability study
identify an optimization criterion
conduct optimization trials
identify the best DHI
conguration that meets the optimization criterion
to conduct a randomized controlled trial
develop a DHI product
monitor reach, impact
and side eects
update the DHI
dierent facets of the usability category relate to dierent
DEDHI framework phases.
Finally, it must be noted, that some of the implemen-
tation barriers could not be aligned to the DEDHI frame-
work as they cannot be overcome during the life cycle of
DHIs but are instead related to framing conditions. This
includes missing benets, cooperation and responsibili-
ties as well as characteristics of the disease involved which
hinder the usage of DHIs in general.
The DEDHI framework provides an overview of evaluation
criteria and implementation barriers to be considered dur-
ing the life cycle phases of DHIs. All criteria and almost
all barriers could be matched to the four phases. However,
all phases could be linked to dierent numbers of crite-
ria and barriers, which underlines the importance of ad-
dressing both factors during the whole life cycle. Further-
more, it underlines the t of the DEDHI framework regard-
ing the purpose of informing DHI developers and evalua-
tors step-wise about criteria and barriers to be considered.
However, dependencies between criteria (e. g., lower rel-
evance of costs whenever a DHI is easy to use and su-
ciently helpful) were not considered in our work as they
could not be identied by the literature review and con-
tent analysis itself.
The evaluation criteria and implementation barriers
presented in this work originate from dierent countries
and geographic regions, for example, the United States
[30], Europe [36, 69], Australia [69] or Africa [34, 85]. This
shows the universality of the criteria and barriers and with
it, also the universality of the DEDHI framework.
Moreover, it becomes obvious from the current work
that the interdisciplinary eld of Digital Health needs to
integrate and consolidate perspectives and research nd-
ings from various elds such as behavioral medicine (e. g.,
the “active” ingredients of DHIs such as well-established
behavior change techniques), computer science (e. g., ma-
chine learning algorithms embedded in DHIs that detect
critical health conditions), software engineering (e.g., the
rigorous design, implementation and test of DHIs) or in-
formation systems research (e.g., understanding the use
and success factors of DHIs). That is, to better understand
the development and evaluation of DHIs, it is crucial to
broaden the scope and to account for related work at the
intersection of the relevant disciplines involved.
Last but not least, no work comes without limitations
which also applies to this one. First and foremost, the
proposed DEDHI framework was developed purely in an
inductive way based on content analysis techniques and
existing justicatory knowledge. It was therefore not ap-
plied, validated and revised during the development and
evaluation of DHIs in the eld. Thus, empirical evidence
that supports the utility of the DEDHI framework is not es-
tablished yet.
Second, the current work considers ndings from sci-
entic outlets only and thus, incorporates country-specic
regulatory frameworks only indirectly to the extent to
which these regulations are covered by these outlets. That
is, legal frameworks and prescriptions with respect to the
life cycle phases will probably dier in detail and depend
on the class of the (medical) DHIs in comparison to the
more idealistic four phases of DEDHI. With the goal to ac-
celerate the digital transformation of health care, for ex-
ample, the German Ministry of Health proposes the imple-
mentation of easy to use and secure DHIs in a rst phase
before their eectiveness is assessed in a second step [86].
This approach has the advantage, in particular for start-
up companies, that signicant nancial investments of up
to several years (e. g., for optimization and evaluation tri-
als) are not required up-front. Instead, in interdisciplinary
collaborations with digital health (research or business)
organizations, relevant stakeholders such as patient orga-
nizations, health insurance or pharmaceutical companies
may take over a signicant amount of these investments
due to the early product character of DHIs. Another advan-
tage is the primary focus on real-world trials compared to
often articial ecacy studies under controlled environ-
ments, for example, with highly selected participants or
study nurses that are experienced with clinical trials [32].
The major shortcoming of such an approach, however, is
the fact that the burden of patients will be increased by
oering DHIs that are (potentially) not eective at all.
Third, the proposed DEDHI framework does not make
an explicit distinction between the goals and motivations
of the various stakeholders interested in the design and
evaluation of DHIs, such as research teams funded by na-
tional research foundations or commercial digital health
companies which are dependent on payers such as health
insurance organizations. While research teams may be pri-
marily interested in the publication of novel digital coach-
ing concepts and their impact on therapy adherence (here,
the focus lies primarily on the preparation phase and op-
timization phase), the primary interest of commercial dig-
ital health companies may be to bring a new DHI as fast
as possible into the healthcare market (here, the focus lies
on the implementation phase). Implications for the doc-
umentation and testing of the DHIs may be very dier-
ent in these cases. For example, the digital health com-
pany must establish well-documented software develop-
ment processes at the very beginning of a new DHI project
as they are hard regulatory requirements when the DHI is
oered in the healthcare market. On the contrary, the very
same regulatory requirements are not relevant for the re-
search team.
And nally, the chosen methods include subjective
procedures. Conducting literature searches and qualita-
tive content analysis is limited by the terms and databases
chosen, and by the subjectivity of the researchers involved.
However, such bias was reduced as much as possible.
For example, relevant databases were included for the
searches and synonyms of search terms were tested for
results. Furthermore, each methodological step was done
by at least two authors independently and inconsistencies
were resolved by discussion and consensus.
Due to the lack of well-established design and assessment
guidelines for digital health interventions (DHIs), the cur-
rent work had the objective to propose a framework for
the Design and Evaluation of DHIs (DEDHI). For this pur-
pose, justicatory knowledge from the elds of behavioral
medicine, medical informatics and information systems
was reviewed. Overall, four life cycle phases of DHIs, 331
evaluation criteria and 98 implementation barriers were
identied and consolidated. The resulting DEDHI frame-
work is meant to support both researchers and practition-
ers alike during the various design and evaluation phases
of DHIs.
Future work is advised to critically apply, reect,
validate, and revise the proposed framework with its
components as the eld of Digital Health is still in its
nascent stage. Accordingly, it is recommended that ex-
perts from the elds of ethics, regulatory aairs, public
health, medicine, computer science and information sys-
tems work closely together to pave the way for evidence-
based DHIs. The latter would not only push the eld of
Digital Health forward but it will, rst and foremost, help
a signicant number of individuals to better manage their
chronic health problems in their everyday lifes.
This work was co-funded by Health Promotion
Switzerland, and the European Social Fund and the Free
State of Saxony (Grant no. 100310385).
Information Systems
Research
JMIR Mhealth
Uhealth
Internet Interventions
Internet
Interventions
Addiction
Planning health promotion programs;
an Intervention Mapping approach
Health Education & Behavior
JAMA
Journal of Medical Internet
Research
Frontiers in Public Health
BMC Health
Services Research
Europen Journal of
Epidemiology
Journal of Telemedicine
and Telecare
Journal of Biomedical Informatics
BMJ
BMJ
Telemedicine Journal and E-Health
Healthcare
Optimization of Behavioral, Biobehavioral,
and Biomedical Interventions: The Multiphase Optimization
Strategy (MOST)
Annals of
Behavioral Medicine
American Journal of
Preventive Medicine
BMJ
Health information from the web – assessing
its quality: a KET intervention
Bulletin of the World
Health Organization
Application of the behaviour change wheel
framework to the development of interventions within the
City4Age project
Journal of Medical Internet Research
Journal of Pediatric Psychology
Population
Health Management
New England Journal of
Medicine
PloS One
Journal of Telemedicine &
Telecare
Health program planning: an
educational and ecological approach
Journal of Psychopathology & Behavioral
Assessment
BMC Health Services Research
Frontiers in Public Health
Bmc Medical Informatics
and Decision Making
BMC Medicine
Health and
Quality of Life Outcomes
JMIR Mhealth Uhealth
Telemedicine & e-Health
Healthcare
Informatics Research
Telemedicine
Journal and E-health
American Journal of Health Promotion
Telemedicine and
e-Health
Health Psychology
A privacy framework for
mobile health and home-care systems
Text-based Healthcare Chatbots Supporting
Patient and Health Professional Teams: Preliminary Results of
a Randomized Controlled Trial on Childhood Obesity
Journal of medical systems
Nature Biotechnology
Forum: Qualitative
Social Research
Implementation Science
JMIR
Public Health & Surveillance
Website quality assessment criteria
Journal of medical Internet Research
American Journal of Preventive
Medicine
Health Psychology
Annals of
Behavioral Medicine
Journal of the American Medical
Informatics Association
Health at a Glance: Europe 2016 – State of Health in
the EU Cycle
Investigating Barriers for the
Implementation of Telemedicine Initiatives: A Systematic
Review of Reviews
JMIR Mhealth Uhealth
Clinical Interventions in Aging
Pain Med
BMJ
International Journal of Medical Informatics
Journal
of Biomedical Informatics
A framework to measure user
experience of interactive online products
Australasian Journal of Information
Systems
Communications of AIS
Australian Journal
of Rural Health
Nature Biotechnology
JMIR
mhealth and uhealth
JMIR
Mhealth Uhealth
JMIR Mhealth
Uhealth
Interactive Journal of
Medical Research
The Lancet
World Psychiatry
International Journal of Environmental Research
and Public Health
Comparative usability
evaluation of a mobile health app
Journal of Epidemiology & Community Health
Studies in Health Technologie
and Informatics
Health expectations: an
international journal of public participation in health care and
health policy
Content uploaded by Tobias Kowatsch
Author content
All content in this area was uploaded by Tobias Kowatsch on Dec 01, 2019
Content may be subject to copyright.
