Content uploaded by Farah Magrabi
Author content
All content in this area was uploaded by Farah Magrabi on Oct 16, 2019
Content may be subject to copyright.
Review
Safety concerns with consumer-facing mobile health
applications and their consequences: a scoping review
Saba Akbar, Enrico Coiera, and Farah Magrabi
Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
Corresponding Author: Farah Magrabi, PhD, Centre for Health Informatics, Australian Institute of Health Innovation, Mac-
quarie University, Level 6, 75 Talavera Road, North Ryde NSW 2113, Sydney, Australia; farah.magrabi@mq.edu.au
Received 29 August 2019; Revised 5 September 2019; Editorial Decision 7 September 2019; Accepted 23 September 2019
ABSTRACT
Objective: To summarize the research literature about safety concerns with consumer-facing health apps and
their consequences.
Materials and Methods: We searched bibliographic databases including PubMed, Web of Science, Scopus, and
Cochrane libraries from January 2013 to May 2019 for articles about health apps. Descriptive information about
safety concerns and consequences were extracted and classified into natural categories. The review was con-
ducted in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-
Analyses extension for Scoping Reviews) statement.
Results: Of the 74 studies identified, the majority were reviews of a single or a group of similar apps (n¼66,
89%), nearly half related to disease management (n¼34, 46%). A total of 80 safety concerns were identified, 67
related to the quality of information presented including incorrect or incomplete information, variation in con-
tent, and incorrect or inappropriate response to consumer needs. The remaining 13 related to app functionality
including gaps in features, lack of validation for user input, delayed processing, failure to respond to health dan-
gers, and faulty alarms. Of the 52 reports of actual or potential consequences, 5 had potential for patient harm.
We also identified 66 reports about gaps in app development, including the lack of expert involvement, poor ev-
idence base, and poor validation.
Conclusions: Safety of apps is an emerging public health issue. The available evidence shows that apps pose
clinical risks to consumers. Involvement of consumers, regulators, and healthcare professionals in development
and testing can improve quality. Additionally, mandatory reporting of safety concerns is needed to improve out-
comes.
Key words: mHealth, mobile applications, consumer health information, patient safety
INTRODUCTION
Advancements in digital technologies have provided consumers with
access to a wide range of resources to manage their health. Health
apps, software programs that run on smartphones and other mobile
communication devices, are an important example because they pro-
vide a variety of different ways to engage and empower consumers.
The numbers of health apps have soared in the last few years, and
by the end of 2017, there were almost 325 000 health apps available
on the leading app stores.
1
Complementary to the rising number of
apps, their demand is also growing. Approximately 3.8 billion apps
were downloaded in 2017, which was a 16% increase from 2016.
1
Health apps provide a range of facilities from simple reminders
and record-keeping diaries to complex medical devices.
2
They are
accessible at all times and they let consumers manage chronic dis-
eases such as diabetes, support lifestyle changes to aid weight loss
and smoking cessation, and even promote self-diagnosis.
3,4
Many
health apps utilize mobile phone features such as cameras and Blue-
tooth to allow users to record behavioral data such as activity and
V
CThe Author(s) 2019. Published by Oxford University Press on behalf of the American Medical Informatics Association.
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/),
which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact
journals.permissions@oup.com 1
Journal of the American Medical Informatics Association, 0(0), 2019, 1–11
doi: 10.1093/jamia/ocz175
Review
Downloaded from https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocz175/5585394 by guest on 16 October 2019
food intake.
5
Such applications make it easier for consumers to
manage their health over time by setting goals and reminders. Apps
typically use techniques such as text messaging, access to personal
health records, and telemedicine or telehealth to engage with their
consumers.
3
They also offer educational resources for consumers
with varying degree of health and digital literacy.
While apps have the potential to benefit consumers by offering
interactive tools that help with treatment adherence and by improv-
ing access to information,
6
they can also pose safety risks if they are
inaccurate and unreliable, mainly because consumers may use the in-
formation from apps to make decisions about their health.
7
Re-
cently, discussions about potential risks and consequences of health
apps have increased, however, there is a lack of consolidated evi-
dence in this area. While the existing literature examines the effec-
tiveness of apps,
8
safety risks are generally discussed as part of other
objectives such as when developing frameworks to assess apps
7,9
and reviewing regulatory implications.
10
Additionally, previous
reviews have mainly focused on the quality of apps that target spe-
cific health conditions, such as diabetes
11
or asthma.
12
A limited
number of studies take a broader look at the risks of using health
apps. For example, one study that reviewed apps used by both
consumers and providers suggested that apps can pose dangers to
physical integrity, bodily well-being, mental well-being, and the pri-
vacy of consumers.
13
To the best of our knowledge, no study has
specifically reviewed safety risks of consumer-facing apps. To ad-
dress this gap, we conducted a scoping review to summarize the re-
search literature about safety concerns with consumer-facing health
apps and their consequences.
MATERIALS AND METHODS
We focused on studies reporting safety concerns with consumer-
facing health apps intended for use primarily by specific patient
groups or general populations.
14
Based on our previous studies,
safety concerns were defined as problems with apps that posed ac-
tual or potential risks of harm to consumers.
15
Search strategy
Bibliographic databases including PubMed, Web of Science, Scopus,
and Cochrane libraries were searched in June 2017 and updated in
May 2019. The search query used was (“safety” OR “risk*” OR
“error*” OR “concern*” OR “problem*” OR “challenge*” OR
“failure” OR “quality”) AND (“app*” OR “application*”) AND
(“smartphone” OR “mobile” OR “mHealth” OR “patient facing”).
Appropriate vocabulary terms were included (Supplementary Ap-
pendix A) and the retrieval set was limited to articles published in
2013 or later.
Study selection
The review was conducted in accordance with the PRISMA-ScR
(Preferred Reporting Items for Systematic Reviews and Meta Analy-
ses extension for Scoping Reviews).
16
After the initial search, dupli-
cate entries and those that were either an erratum or response to
another published article were removed (Figure 1). The titles and
abstracts of the remaining articles were screened by a single reviewer
to identify relevant studies. Study designs were limited to analyses of
apps, pilot tests, randomized controlled trials, and systematic
reviews. Non-English articles and conference abstracts were ex-
cluded. Full-length articles were retrieved from all abstracts identi-
fied for inclusion and were assessed independently against the
inclusion criteria by two reviewers (SA and FM). Articles that did
not meet any of the inclusion criteria were excluded and any dis-
agreements about inclusion or exclusion of an article were resolved
by consensus.
Data extraction and synthesis
For each included study, descriptive information about the apps
studied, study design, sample size, safety concerns, and consequen-
ces were extracted.
Type of apps
The type of apps was examined using the U.S. Food and Drug Ad-
ministration (FDA) classification (high-risk mobile medical apps,
low-risk mobile medical apps, nonmobile medical apps). The FDA
regulates high-risk medical apps and exercises enforcement discre-
tion for low-risk medical apps.
17
Health domains
Health domains addressed by apps were examined using the catego-
ries of consumer mobile health apps described by Kao and Liebovitz
(wellness management, disease management, self-diagnosis, medica-
tion reminder, physical medicine and rehabilitation).
18
Consumer engagement strategy
Consumer engagement was examined using categories from the
framework developed by Singh et al (providing educational informa-
tion; reminding or alerting users; recording and tracking health in-
formation; displaying and summarizing health information;
providing guidance based on information entered by the user; en-
abling communication with clinicians, family members, and care-
givers; providing support through social networks; supporting
behavior change through rewards).
19
Categorization of safety concerns
Using a grounded approach both reviewers iteratively examined de-
scriptive information about safety concerns to identify natural cate-
gories relating to the quality apps themselves and the processes
undertaken to develop them.
20
Consequences
Consequences were categorized using a standard approach into the
following
15
:
1. Potential or actual harm to a consumer: A safety concern that
reached consumers (eg, an app yielded a wrong dose for insulin
that the user followed and became hypoglycemic).
2. An arrested or interrupted sequence or a near miss: A safety con-
cern that was detected before reaching consumers (eg, a con-
sumer recognized the incorrect recommendation in the app and
did not follow it).
3. Noticeable consequence but no harm: A concern that affected
use of the app but no harm to consumers (eg, time wasted wait-
ing for an app to function correctly).
4. No noticeable consequence: A concern that did not directly af-
fect safe use of the app (eg, consumers did not like the font used
to present information in the app but did not stop using it).
5. Hazardous event or circumstance: A concern that could poten-
tially lead to an adverse event or a near miss (eg, an app under-
diagnosed a malignant skin lesion, falsely reassuring consumers).
2Journal of the American Medical Informatics Association, 2019, Vol. 0, No. 0
Downloaded from https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocz175/5585394 by guest on 16 October 2019
The two reviewers independently examined free-text event
descriptions of safety concerns to assess consequences. Interrater re-
liability analysis using the kappa statistic was performed to deter-
mine consistency among reviewers. When reviewers disagreed, the
report was re-examined and a consensus category assigned. The
interrater reliability (kappa) for the classification was 0.92
(P<.001; 95% CI; 0.77-1.0). A narrative synthesis then integrated
findings into descriptive summaries for each category of safety con-
cerns.
RESULTS
Our search returned a total of 3456 titles and abstracts. After re-
moval of duplicate entries and errata, 2388 abstracts were screened.
Of these, 2314 studies were excluded for various reasons leaving 74
studies that provided reports about safety concerns of mobile health
apps (Figure 1).
Descriptive analysis of selected studies
Of the 74 studies included in our review, 17 (23%) were published
in 2018 (Table 1;Supplementary Appendix B). The majority were
reviews of apps (n ¼66, 89%), which evaluated the quality of con-
tent and functionality of either a single app or a group of apps tar-
geting a specific audience, for example, apps for diabetes
management.
3
The number of apps reviewed in these studies typi-
cally ranged between 1 and 756. Only 11 of these reviews examined
apps in the hands of consumers. Four of the included studies were
literature reviews about groups of apps,
21–23
3 were randomized
controlled trials,
24–26
and 1 was a nonrandomized controlled trial
27
Type of app, domain and consumer engagement
strategy
We examined the type of apps using the FDA classification.
17
Most
studies related to low-risk medical devices (n ¼42, 57%) or wellness
(n ¼25, 34%). Seven (9%) related to high-risk medical device apps
including apps for monitoring vital signs,
23,28
and diagnosis of dis-
ease including melanoma
29–32
and color blindness.
33
Almost half of the studies related to apps for managing a specific
disease (n ¼34, 46%) with the remaining focusing on apps for well-
ness management, self-diagnosis, physical medicine, and medication
reminders (Table 1;Supplementary Appendix C). Most studies in-
volved apps that engaged consumers using 1 or more functionalities
such as by displaying and summarizing user-entered information
(n ¼54), tracking data (n ¼52), providing guidance (n ¼49), and
educational information (n ¼49).
Safety concerns
Our examination of the 74 studies revealed 80 safety concerns that
were extracted and categorized. The vast majority of safety concerns
related to quality of the content presented in apps (n ¼67, 84%).
Only 13 safety concerns (16%) related to software functionality,
which is the match of the app user interface and functions to con-
sumer engagement strategies.
Quality of content
The 67 safety concerns relating to the quality of information pre-
sented by apps could be grouped into 5 categories including incor-
rect information, incomplete information, variation in content, and
incorrect and inappropriate response to consumer needs (Box 1). In
the following list, we give a brief overview of each of these catego-
ries.
1. Incorrect information: Reports about wrong information being
presented by apps were common.
22,24,34–47
For example, apps
for bipolar disorder (BD) incorrectly
47
differentiated between
BD types, and wrongly recommended that patients should “take
a shot of hard liquor an hour before bed.”
40
The same group of
apps also suggested that BD is contagious (ie, it can be spread to
anyone who spends a lot of time with a BD patient).
38,40
An-
Figure 1. Article search and retrieval process.
Journal of the American Medical Informatics Association, 2019, Vol. 0, No. 0 3
Downloaded from https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocz175/5585394 by guest on 16 October 2019
other study reported that urolithiasis apps recommended
lowering calcium intake, which was incorrect and contradicted
the available evidence.
38
2. Incomplete information: Apps were also reported to provide in-
complete information to consumers.
32,36,46–56
For example, in-
sulin dose calculation apps did not offer a mechanism for
reducing amount of postmeal administration that was required
because the body may produce its own residual insulin
50
and
breast cancer apps were silent about hormonal receptors and the
new classification based on the HER (human epidermal
receptor).
36
Moreover, apps were also found to miss critical in-
formation about health conditions. For example, one study
found that apps to diagnose pigmented lesions did not offer clear
recommendations about what consumers should do when they
receive a provisional negative diagnosis for their lesion.
32
Simi-
larly, many apps that support nutrition allowed users to set die-
tary goals but did not provide guidance on appropriate goal
setting.
55
Another study that assessed the content of the top 5 cardiovascu-
lar apps against European guidelines found that only 1 app con-
tained 6 of 8 key topic areas. Surprisingly, none of these apps
addressed regular medical follow-up and smoking cessation,
which were identified as key topics in the guidelines.
51
3. Variation in content: Apps that addressed similar domains were
found to have significant differences in the quality of their con-
tent.
33,37,47,57–61
For example, studies reported inconsistencies
in the information presented and tools used in apps for obesity
management,
58
physical activity measurement,
59
color vision as-
sessment,
33
and medication self-management.
60
Likewise, apps
for pain management required users to enter varying amounts of
information to clinically assess symptoms.
57
4. Incorrect output: Apps that provided calculations and diagnostic
outputs were reported to be incorrect.
23,24,28–32,37,44,50,64,66–69
For example, blood alcohol concentration (BAC) apps overesti-
mated the amount of alcohol in the breath or blood
24,37
and
melanoma detection apps incorrectly diagnosed melanomas and
benign lesions.
29–31
In addition, some apps were reported to
carry the risk of generating wrong outputs if the user enters inva-
lid information such as a negative value in insulin calculation
apps,
50
or if the user’s finger position was incorrect in apps that
scanned fingertips to monitor heart rate.
67
5. Inappropriate response to consumer needs: Many apps provided
facilities for consumers to enter information about their physical
and mental health status, so they could be guided accordingly.
67
Some of these apps responded inappropriately,
63,65
especially
when users reported feeling suicidal or entered extremely
high scores for shortness of breath or pain.
65
Such apps failed to
encourage consumers to seek professional help at critical
times.
63
Software functionality
The 13 concerns relating to the functionality of apps could be
grouped into 5 other categories including gaps in features, lack of
validation for user input, delayed processing, failure to respond to a
health danger, and faulty alarms (Box 2). In the following, we give a
brief overview of each of these categories.
1. Gaps in features: Many studies found that apps did not ade-
quately support consumer tasks.
52,53,60,70–72
For example, some
medication self-management apps did not support oral contra-
ceptives, medications to be taken as needed (PRN), over-the-
counter drugs, and variable-dose medications.
60
Others did not
allow users to enter dose in grams instead of milliliters or to cus-
tomize their dosing schedule.
72
Similarly, alcohol cessation apps
lacked important features for motivation, identification of risk
situations and coping strategies for relapse.
70
2. Lack of validation for user input: Validation of the data entered
by users, which is an important first step, was reported to be
missing in many apps. For example, insulin dose calculation
apps did not have facilities for simple numeric validation to pre-
vent missing values and text entries in the fields intended for
users to enter blood glucose values.
50
These apps were reported
to allow calculations despite missing 1 or more values. In other
cases, apps did not allow users to change values that had been
entered incorrectly.
26
3. Delayed processing: There were reports about the time taken by
apps to process information and generate outputs which could
critically affect consumer safety. For example, vital signs moni-
toring apps, which are considered among high risk mobile medi-
cal apps, were reported to measure ECG with a delay of 30-60
seconds.
67
4. Response to health dangers: Apps were reported to be unrespon-
sive to safety-critical information entered by users. For instance,
bipolar disorder apps failed to provide any response when infor-
Table 1. Characteristics of studies reporting safety risks of mobile
health apps (N ¼48)
Characteristics n %
Study design
review of app(s) 66 89
literature review 4 5
randomized controlled trial 3 4
nonrandomized controlled trial 1 1
Year of publication
2013 6 8
2014 8 11
2015 11 15
2016 15 20
2017 14 19
2018 17 23
2019 (until May) 3 4
Types of apps
High-risk medical device 7 9
Low-risk medical device 42 57
Wellness 25 34
Consumer engagement functionalities
a
provides educational information 49 66
reminds or alerts patient 19 26
tracks information 52 70
displays and summarizes user-entered information 54 73
provides guidance based on user-entered information 49 66
enables communication with family or clinician 14 19
provides support through social networks 20 27
rewards behavior change 14 19
Domain
18
disease management 34 46
wellness management 26 35
self-diagnosis 6 8
medication reminder 2 3
physical medicine and rehabilitation 5 7
all health domains 1 1
a
Categories are not mutually exclusive. One study may include apps using
multiple engagement functionalities.
4Journal of the American Medical Informatics Association, 2019, Vol. 0, No. 0
Downloaded from https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocz175/5585394 by guest on 16 October 2019
mation about severe extremes of mood or suicidal ideation was
entered.
40
5. Faulty alarms: One of the basic consumer engagement function-
alities for health apps are reminders, which are usually generated
in the form of alarms.
19
Two studies of apps for medication self-
management found problems with in-built alarms.
60,72
Consequences of safety concerns
There were 52 reports about the actual or potential consequences of
safety concerns.
21,22,24,25,27,29–38,41,42,44,47–52,66,68,72–80
Five had po-
tential for patient harm. In one study, use of a BAC calculation app
led to a significant increase in alcohol consumption among a
group of 341 students with risky alcohol use (P<.001). At 7 weeks,
participants with the baseline drinking frequency of 2.24 occasions/
week reported an increase to 2.36 occasions/wk, while the control
group reported a decrease from 9.15 to 8.62 glasses/wk.
74
Similarly,
users of an app providing a standalone intervention to reduce dis-
tress and alcohol consumption reported an increase in typical drink-
ing and heavy drinking.
27
Those who used the app for more than 4
weeks also reported increased anxiety and distress.
27
In another
study, which examined an app to support women undergoing breast
cancer surgery, postoperative anxiety, and depression, scores were
significantly higher than the in control group (n ¼13; P¼.029 and
.022, respectively).
25
While there were no reports about near miss events, 4 safety con-
cerns had noticeable consequences (8%). A review of apps for type 2
Box 1. Example safety concerns relating to content quality
Incorrect information: An app for sexually transmitted infections suggested that “Genital warts are bad. If they form in a
bunch on your genitals, you will have a very bad time getting them treated and your relationships will shatter.” Another
app noted that, “Candida (found in yeast infections) can infect your blood, causing an overload of toxins to disrupt your sys-
tem, wreaking havoc on your mind and body.”
62
Fetal heart rate monitoring apps provided incorrect statements about nor-
mal heart rate, heart rate differences between genders and warnings such as hot foods and rinds of papayas causing mis-
carriage.
44
Incomplete information: Exercise apps lacked information about indication, frequency, or description of performing the rec-
ommended activity.
41,46
Another study found that only 4 of 33 (12%) depression apps provided guidance regarding crisis
management.
63
Variation in content: Apps that provide guidance about decreased fetal movement had varying information about normal
frequency, with some suggesting that 10 kicks felt in 2 hours should be reassuring, while others suggested 10 movements
experienced over 12 hours or 1 hour was normal.
47
Incorrect output: Apps to monitor heart rate produced incorrect measurements with absolute differences of over 20 beats/
min
64
; melanoma risk assessment apps underdiagnosed potentially life-threatening melanomas.
30
In another study, Blood
Alcohol Concentration (BAC) apps were found to overestimate BAC levels by approximately 3 times.
24
The Instant Blood
Pressure app, which estimates blood pressure using the patient’s index finger and positioning along the chest wall, could
not detect hypertensive blood pressure ranges
23
Inappropriate response to consumers’ needs: Of the 121 apps targeting high-risk, high-cost populations, that allow patients
to record health-oriented information, only 28 (23%) responded appropriately when information was entered that indicated a
health danger, such as suicidal mood or ideation.
12 65
In another study, 21 of 33 apps for depression did not include content
aimed at encouraging professional help seeking when needed.
63
Box 2. Example safety concerns relating to software functionality
Gaps in features: Teledermatology apps did not account for allergies or current medication status, both of which could affect
the behavior of the app.
52
Lack of validation for user input Apps for insulin dose calculation lacked standard terminology and simple numeric valida-
tion that provided latent conditions, increasing the risk of unintentional data entry slips and mistakes related to misunder-
standing.
50
A teledermatology app that generated pick lists for users did not contain required option. In one instance, the user wanted
to indicate allergy to sulfa drugs generally but was forced to choose specific ones listed.
52
Delayed processing: Smartphone-based electrocardiography measurement had a delay of 30-60 seconds.
67
Response to health dangers: While apps asked users to input personal health data, very few responded to indications that
users were unwell. In fact, only 3 of 35 symptom monitoring apps responded to users indicating severe extremes of mood
or suicidal ideation.
40
Faulty alarms: Alarms and notifications in the medication self-management apps were either too loud or not loud enough,
did not provide snooze options, or failed to work when phone screens were off.
60
,
72
Journal of the American Medical Informatics Association, 2019, Vol. 0, No. 0 5
Downloaded from https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocz175/5585394 by guest on 16 October 2019
diabetes risk assessment reported that false positives generated by
unvalidated risk scores could overwhelm services.
66
Similarly, apps
for melanoma risk assessment were reportedly overdiagnosing
benign nevi, leading to an unnecessary drain on dermatology resour-
ces.
30
Errors in insulin dosage calculator apps
50
and false readings
from fetal heart monitoring apps
44
were associated with an increase
in the use of unscheduled care by patients.
50
Three safety concerns did not have a noticeable consequence on
care delivery (6%). Participants of a study comparing the Ishihara
booklet with 2 color vision apps rated both the apps lower on com-
fort and clarity.
33
Likewise, a another study that claimed to diag-
nose skin lesions reported that the general population was likely to
use the app for all the lesions that they find suspicious, because it
was difficult for users to distinguish between benign and cancerous
lesions.
31
Another study of cardiovascular apps reported that users
could be misinformed due to substandard information quality.
51
While these findings are notable, there were no consequences
reported.
Forty (77%) safety concerns were associated with potentially
hazardous circumstances. The most frequently reported hazard was
the apps’ potential to mislead users by presenting information
that was neither evidence based nor endorsed by medical
experts.
24,25,36–38,48,72–80
Such misleading information or absence of
critical information was also reported to potentially worsen users’
health conditions and cause indirect harm.
22,24,34,37,41,47,48,50,66,74,77
34,37,41,47,48,50,66,74,77
For example, the study of BAC calculation
apps reported that users were provided with information about how
much more alcohol they could consume before their driving ability
was compromised. Such features may encourage alcohol consump-
tion.
22,24,37
In one study that evaluated sports coaching apps, researchers
found that 23 of 30 apps did not provide instructions about how to
choose a workout or how to organize them over a week. The ab-
sence of this information could adversely affect users who did not
have the appropriate level of preparedness required for each work-
out.
41
Similarly, misleading information about fetal movements
could result in missed opportunities to prevent adverse outcomes
such as stillbirth.
47
In addition, another study found that urolithiasis
apps recommended consumption of a low-calcium diet, which con-
tradicted evidence and had been shown to be harmful.
38
Another commonly reported hazard was linked with incorrect
diagnostic output that could result in false reassurance or unneces-
sary anxiety. Three studies that evaluated apps for melanoma detec-
tion reported incorrect classification of cancerous lesions as “un-
concerning.”
29,31,68
Similarly, apps for diabetes risk assessment
reported false negative outputs.
66
If the visit to medical professional
was substituted by use of such apps, users may be falsely reassured
increasing the risks of harm.
29,31,32,66
Conversely, false positives or
higher disease risk scores were reported to increase user anx-
iety.
44,66,68,77
Gaps in app development
Among the included studies, there were 66 reports about gaps in
processes to design and build apps, including the lack of expert in-
volvement, evidence base, and validation. In the following, we give
a brief overview of each of these categories.
1. Lack of expert involvement: Many studies found that there was
a lack of involvement of subject matter experts in content devel-
opment.
22,29,36,38,42,46,51,55–57,73,75,78–85
For example, only a
limited number of apps relating to major vascular diseases,
73
urology,
38,81
Alzheimer’s and related dementia,
56
mental health
disorders,
22
and chronic pain
57
were developed in consultation
with clinical experts or recognized healthcare agencies or organi-
zations.
82
2. Not evidence based: Many studies found that app content was not
based on the available evidence, adequately referenced, updated
to reflect current evidence, or offered information that contra-
dicted the available evidence.
22,35,36,38,39,42,43,46,47,49,56,63,66,71,75–
78,85–92
For example, a study that evaluated asthma management
apps reported that of the 8 apps that presented recommendation
about removal of pets from home, only 1 was consistent with evi-
dence.
39
Another study of apps that support bariatric surgery or
weight loss surgery patients did not provide references for educa-
tional information.
86
Only around 10% of depression apps in-
cluded evidence-based principles.
22
Moreover, a study of exercise
apps reported that these apps were not following evidence-based
principles set forth by the American College of Sports Medicine.
49
Additionally, in a study of apps that target cancer patients, only 51
of 166 (30%) apps had been updated in the last 2 years. This study
alsofoundthatthecontentofsomebreastcancerappswasobso-
lete.
36
3. Poor validation: Lack of formal validation, which is an
important indicator of the safety of diagnostic, screening,
and assessment tools used within apps, was commonly
reported.
21,22,35,39,40,50,59,78,80,82,93,94
For example, calculators,
questionnaires and assessment tools in asthma management
apps had not been formally tested.
39
The same group of apps
also contained experimental screening products that had not re-
ceived regulatory approval.
39
In another study of apps for main-
taining a diary of headache, none of the 38 apps reviewed had
been subject to formal testing of psychometric properties
94
DISCUSSION
While health apps have the potential to provide easy and low-cost
access to care, little is known about the types of safety concerns as-
sociated with their use.
95
Previous studies have mainly reviewed a
limited number of apps
96
or focused on specific areas of risk such as
privacy.
97
Our review is the first to summarize the kinds of clinical
safety concerns with consumer-facing apps and their consequences.
We identified 10 natural categories relating to the quality of apps
themselves and the processes undertaken to develop them. Gaps in
processes to design and build apps including the lack of expert in-
volvement, evidence base, and validation were also identified.
We found that health apps pose risks to consumer safety when the
content presented within apps is inappropriate or software functional-
ity is compromised. Both these components hold equal significance, as
weakness in one can negatively affect the other. The 10 categories of
safety concerns that we identified were linked with consequences
ranging from actual harm to hazardous events. Based on our findings,
we make a number of recommendations for app developers, health-
care professionals, regulators, consumers, and researchers.
Developers should take a user-centered approach, involving sub-
ject matter experts and consumers in app development.
98,99
We
found that app development processes significantly lack the involve-
ment of relevant healthcare professionals or agencies. This finding is
consistent with previous reviews.
100,101
Experts such as clinicians,
technicians, nurses, pharmacists, and therapists possess the right sets
of knowledge and skills to lead information design. They have been
trained professionally to manage health of people and are usually
aware of the current medical guidelines. Moreover, they interact
6Journal of the American Medical Informatics Association, 2019, Vol. 0, No. 0
Downloaded from https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocz175/5585394 by guest on 16 October 2019
with consumers regularly and are better aware of their concerns.
Hence, their absence from the process can lead to poor quality of
content. They should be involved at 3 stages
102
: (1) the development
phase, when app developers are compiling content for their apps; (2)
the internal validation phase, when the information and tools in-
cluded in apps are validated to confirm if they are correct; and (3)
the verification phase, when apps are tested to check if they perform
as expected.
Consumers are another group that should be engaged in app de-
velopment, particularly in usability testing. Similar to the previous
studies of self-management apps,
103,104
our review indicates that
consumers were able to recognize many critical issues with apps,
such as incorrect information, inappropriate response to their needs,
gaps in features, and faults with alarms. This suggests that involve-
ment of consumers in usability testing will allow problems to be
identified and resolved before apps are published. Usability testing
allows users to provide important insights about app functionality
and medical reliability,
105
helps determine whether the app is conve-
nient for users to perform required tasks, and can reduce costs of fix-
ing errors that may be identified later.
106
Hence, usability testing
offers a win-win situation for both app developers and consumers.
In the postdevelopment stage, apps need to be kept up to date to
reflect current evidence and should be routinely audited.
95
We found
that apps either lacked current evidence or offered information that
contradicted the evidence. With frequent updates to the evidence,
app developers should carefully plan updates to ensure apps are up
to date.
As for healthcare professionals, they should get involved in app
development, promotion, and evaluation. Evidence suggests that the
majority of providers are open to apps
107,108
but hesitate in promot-
ing them, mainly because of the difficulty in identifying apps that
are effective,
109
shortage of time, legal issues, and data security and
privacy concerns.
110,111
To improve the situation, professionals can
participate in app development processes and collaborate with other
providers to perform app evaluation studies. They can also review
scientific literature
112
and have productive discussions with patients
about their usability preferences,
110
so as to help patients make in-
formed decision about use of apps.
Regulators need to actively monitor and address safety concerns.
In the United States, high-risk health apps undergo regulatory
checks by the FDA, Office of Civil Rights, and the Federal Trade
Commission for efficacy, information protection, and security
breaches, respectively.
65
Other national and regional bodies in Aus-
tralia,
113
United Kingdom,
114
New Zealand,
115
and Catalonia and
Andalusia in Spain
116
are also working to formulate guidelines for
app regulation. However, there is a need to develop a monitoring
framework that allows consumers to report safety concerns about
apps and assists with their management. One such example is the
newly announced digital health software precertification program
by the FDA through its real-world performance monitoring strat-
egy.
117
Through this program, the FDA plans to collect information
about consumers’ experience, software performance, and clinical
outcomes and address emerging risks.
117
Consumers need to make more informed choices about apps. At
present, consumers encounter hundreds of thousands of health apps
when they search app stores. While most consumers prefer using
apps that are recommended by their providers or peers,
118
many are
still on their own when making the choice. To make an informed de-
cision that is safe and trustworthy, it is important that consumers
carefully read descriptions and outcome reports about apps that
they are considering as well as search for app developers’ credibility.
Relying on the app store rating is not recommended.
105,119
Curated
libraries of apps established by trusted sources, such as National
Health Service UK, can be resourceful tools for consumers to navi-
gate through safe apps.
120
Other proposed strategies such as grading
labels may also be useful.
121
The leading app stores encourage users to report any inappropri-
ate content or functionality. Consumers should use these platforms
to report any safety issues that they encounter while using apps. Ad-
ditionally, they can also give their feedback to app developers and
regulatory agencies such as the FDA.
122
More primary studies on app safety are required. There is plenty
of literature available on health apps. During our search, we
screened 2388 abstracts, of these, more than half studied apps.
However, only a handful of studies engaged consumers and allowed
them to express their concerns relating to the safety of apps. More-
over, to the best of our knowledge, there is no standard method of
reporting safety concerns in app testing. There are general frame-
works that cover health information technology, such as CON-
SORT-eHealth (Consolidated Standards of Reporting Trials of
Electronic and Mobile Health Applications and Online Tele-
Health)
123
and STARE-HI (Statement on reporting of evaluation
studies in Health Informatics),
124
but these are limited to specific
study types and do not mandate reporting of safety concerns. The
reporting of safety concerns and consequences should be mandatory
to encourage researchers to evaluate and report safety concerns with
apps. Studies are also required to examine the magnitude of the
harm from health apps.
LIMITATIONS
This review was restricted to the published literature. The concerns
were identified from studies that used a wide variety of designs and
may not have captured all possible issues with the different types of
apps that are currently available to patients. We did not include the
grey literature or any other source of information about apps such
as user complaints, and app store reviews. Another limitation is the
screening of titles and abstracts was performed by a single reviewer.
Studies that noted safety concerns only in the results or discussion
sections may have been missed. We also excluded non-English
articles, which limits the generalizability of our findings to apps that
target non-English speakers.
CONCLUSIONS
Health apps may have significant potential to improve population
health. However, to ensure that this potential is met, it is important
that apps are safe, effective, and reliable. The gaps in app develop-
ment, safety concerns, and consequences found in this review call
for increased stakeholder engagement, vigilant regulatory frame-
works, and more focused research. The reporting of safety concerns
and consequences should be mandated in reporting guidelines. These
improvements will build trust and increase the confidence of both
providers and consumers.
FUNDING
This research is supported by the Australian National Health and
Medical Research Council Centre for Research Excellence in Digital
Health grant 1134919 (FM and EC). The funding source did not
play any role in study design, in the collection, analysis, and inter-
Journal of the American Medical Informatics Association, 2019, Vol. 0, No. 0 7
Downloaded from https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocz175/5585394 by guest on 16 October 2019
pretation of data, in the writing of the report, or in the decision to
submit the article for publication.
AUTHOR CONTRIBUTIONS
FM and EC conceptualized the study. SA and FM led the literature
search, data analysis, and drafted the article. SA is responsible for
the integrity of the work. She is the guarantor. All authors partici-
pated in writing and revising the article. All aspects of the study
(including design; collection, analysis and interpretation of data;
writing of the report; and decision to publish) were led by the
authors.
SUPPLEMENTARY MATERIAL
Supplementary material is available at Journal of the American
Medical Informatics Association online.
ACKNOWLEDGMENTS
We thank Jessica Chen for assisting with full text screening of the updated
search.
CONFLICT OF INTEREST
None declared.
REFERENCES
1. Joshi A, Generalla J, Thompson B, Haidet P. Facilitating the feedback
process on a clinical clerkship using a smartphone application. Acad Psy-
chiatry 2017; 41 (5): 651–5.
2. Wicks P, Chiauzzi E. Trust but verify’–five approaches to ensure safe
medical apps. BMC Med 2015; 13: 205.
3. Boulos MN, Brewer AC, Karimkhani C, Buller DB, Dellavalle RP. Mo-
bile medical and health apps: state of the art, concerns, regulatory con-
trol and certification. Online J Public Health Inform 2014; 5 (3): 229.
4. Peng W, Kanthawala S, Yuan S, Hussain SA. A qualitative study of user
perceptions of mobile health apps. BMC Public Health 2016; 16 (1):
1158.
5. Pellegrini CA, Duncan JM, Moller AC, Buscemi J, Sularz A, DeMott A.
A smartphone-supported weight loss program: design of the ENGAGED
randomized controlled trial. BMC Public Health 2012; 12 (1): 1041.
6. Mcaskill R. The benefits of mobile health strategies: mHealth Intelli-
gence; 2015. https://mhealthintelligence.com/news/the-benefits-of-mo-
bile-health-strategies. Accessed August 22, 2018.
7. Lewis TL, Wyatt JC. mHealth and mobile medical apps: a framework to
assess risk and promote safer use. J Med Internet Res 2014; 16 (9): e210.
8. Byambasuren O, Sanders S, Beller E, Glasziou P. Prescribable mHealth
apps identified from an overview of systematic reviews. npj Digit Med
2018; 1 (1): 12.
9. Nouri R, Kalhori SRN, Ghazisaeedi M, Marchand G, Yasini M. Criteria
for assessing the quality of mHealth apps: a systematic review. J Am Med
Inform Assoc 2018; 25 (8): 1089–98.
10. Terry NP, Gunter TD. Regulating mobile mental health apps. Behav Sci
Law 2018; 36 (2): 136–44.
11. Sun C, Malcolm JC, Wong B, Shorr R, Doyle MA. Improving glycemic
control in adults and children with type 1 diabetes with the use of
smartphone-based mobile applications: a systematic review. Can J Dia-
betes 2019; 43: 51–8.e3.
12. Farzandipour M, Nabovati E, Sharif R, Arani MH, Anvari S. Patient
self-management of asthma using mobile health applications: a system-
atic review of the functionalities and effects. Appl Clin Inform 2017;
8 (4): 1068–81.
13. Albrecht UV, Von Jan U. mHealth apps and their risks - taking stock.
Stud Health Technol Inform 2016; 226: 225–8; IOS Press.
14. IMS Institute for Healthcare Informatics. Patient adoption of mHealth.
IMS Institute for Healthcare Informatics; 2015. https://www.iqvia.com/-
/media/iqvia/pdfs/institute-reports/patient-adoption-of-mhealth.pdf.
Accessed October 02, 2018.
15. Kim MO, Coiera E, Magrabi F. Problems with health information tech-
nology and their effects on care delivery and patient outcomes: a system-
atic review. J Am Med Inform Assoc 2017; 24 (2): 246–50.
16. Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D.
PRISMA extension for scoping reviews (PRISMA-ScR): checklist and ex-
planation. Ann Intern Med 2018; 169 (7): 467–73.
17. Mobile medical applications: U.S. Food and Drug Administration; 2018.
https://www.fda.gov/MedicalDevices/DigitalHealth/MobileMedicalAp-
plications/default.htm#b. Accessed August 12, 2019.
18. Kao CK, Liebovitz DM. Consumer mobile health apps: current state,
barriers, and future directions. PM R 2017; 9 (5S): S106–S15.
19. Singh K, Drouin K, Newmark LP, et al. Developing a framework for
evaluating the patient engagement, quality, and safety of mobile health
applications. Issue Brief (Commonw Fund) 2016; 5: 1–11.
20. Runciman WB, Edmonds MJ, Pradhan M. Setting priorities for patient
safety. Qual Saf Health Care 2002; 11 (3): 224–9.
21. Eng DS, Lee JM. The promise and peril of mobile health applications for
diabetes and endocrinology. Pediatr Diabetes 2013; 14 (4): 231–8.
22. Magee JC, Adut S, Brazill K, Warnick S. Mobile app tools for identifying
and managing mental health disorders in primary care. Curr Treat
Options Psychiatry 2018; 5 (3): 345–62.
23. Sarwar CMS, Vaduganathan M, Anker SD, et al. Mobile health applica-
tions in cardiovascular research. Int J Cardiol 2018; 269: 265–71.
24. Weaver ER, Horyniak DR, Jenkinson R, Dietze P, Lim MS. “Let’s get
wasted!” and other apps: characteristics, acceptability, and use of
alcohol-related smartphone applications. JMIR mHealth Uhealth 2013;
1 (1): e9.
25. Foley NM, O’Connell EP, Lehane EA, et al. PATI: Patient accessed tai-
lored information: a pilot study to evaluate the effect on preoperative
breast cancer patients of information delivered via a mobile application.
Breast 2016; 30: 54–8.
26. Skar JB, Garnweidner-Holme LM, Lukasse M, Terragni L. Women’s
experiences with using a smartphone app (the Pregnantþapp) to manage
gestational diabetes mellitus in a randomised controlled trial. Midwifery
2018; 58: 102–8.
27. Hamamura T, Suganuma S, Ueda M, Mearns J, Shimoyama H. Stand-
alone effects of a cognitive behavioral intervention using a mobile phone
app on psychological distress and alcohol consumption among Japanese
Workers: pilot nonrandomized controlled trial. JMIR Ment Health
2018; 5 (1): e24.
28. Alexander JC, Minhajuddin A, Joshi GP. Comparison of smartphone
application-based vital sign monitors without external hardware versus
those used in clinical practice: a prospective trial. J Clin Monit Comput
2017; 31 (4): 825–31.
29. Wolf JA, Moreau JF, Akilov O, et al. Diagnostic inaccuracy of smart-
phone applications for melanoma detection. JAMA Dermatol 2013; 149
(4): 422–6.
30. Chadwick X, Loescher LJ, Janda M, Soyer HP. Mobile medical appli-
cations for melanoma risk assessment: false assurance or valuable
tool? In: Sprague RH, ed. 2014 47th Hawaii International Conference
on System Sciences. Proceedings of the Annual Hawaii International
Conference on System Sciences. New York, NY: ACM; 2014:
2675–84.
31. Nabil R, Bergman W, Kukutsch NA. Poor agreement between a mobile
phone application for the analysis of skin lesions and the clinical
diagnosis of the dermatologist, a pilot study. Br J Dermatol 2017; 177
(2): 583–4.
32. Ngoo A, Finnane A, McMeniman E, Tan JM, Janda M, Soyer HP. Effi-
cacy of smartphone applications in high-risk pigmented lesions. Aus-
tralas J Dermatol 2018; 59 (3): e175–82.
8Journal of the American Medical Informatics Association, 2019, Vol. 0, No. 0
Downloaded from https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocz175/5585394 by guest on 16 October 2019
33. Sorkin N, Rosenblatt A, Cohen E, Ohana O, Stolovitch C, Dotan G.
Comparison of Ishihara booklet with color vision smartphone applica-
tions. Optom Vis Sci 2016; 93 (7): 667–72.
34. Dubad M, Winsper C, Meyer C, Livanou M, Marwaha S. A systematic
review of the psychometric properties, usability and clinical impacts of
mobile mood-monitoring applications in young people. Psychol Med
2018; 48 (2): 208–28.
35. Brouard B, Bardo P, Bonnet C, Mounier N, Vignot M, Vignot S. Mobile
applications in oncology: is it possible for patients and healthcare
professionals to easily identify relevant tools? Ann Med 2016; 48 (7):
509–15.
36. Collado-Borrell R, Escudero-Vilaplana V, Ribed-Sanchez A, Ibanez-Gar-
cia S, Herranz-Alonso A, Sanjurjo-Saez M. Smartphone applications for
cancer patients; what we know about them? Farm Hosp 2016; 40 (1):
25–35.
37. Wilson H, Stoyanov SR, Gandabhai S, Baldwin A. The quality and accu-
racy of mobile apps to prevent driving after drinking alcohol. JMIR
mHealth Uhealth 2016; 4 (3): e98.
38. Stevens DJ, McKenzie K, Cui HW, Noble JG, Turney BW. Smartphone
apps for urolithiasis. Urolithiasis 2015; 43 (1): 13–9.
39. Huckvale K, Morrison C, Ouyang J, Ghaghda A, Car J. The evolution of
mobile apps for asthma: an updated systematic assessment of content
and tools. BMC Med 2015; 13: 58.
40. Nicholas J, Larsen ME, Proudfoot J, Christensen H. Mobile apps for bi-
polar disorder: a systematic review of features and content quality. J
Med Internet Res 2015; 17 (8): e198.
41. Modave F, Bian J, Leavitt T, Bromwell J, Harris C, Vincent H. Low qual-
ity of free coaching apps with respect to the American College of Sports
Medicine Guidelines: a review of current mobile apps. JMIR mHealth
Uhealth 2015; 3 (3): e77.
42. Robertson GAJ, Wong SJ, Brady RR, Subramanian AS. Smartphone
apps for spinal surgery: is technology good or evil? Eur Spine J 2016; 25
(5): 1355–62.
43. Lambert K, Mullan J, Mansfield K, Owen P. Should we recommend renal
diet–related apps to our patients? An evaluation of the quality and health
literacy demand of renal diet–related mobile applications. J Ren Nutr
2017; 27 (6): 430–8.
44. Soffer MD, Chen KT. In search of accurate fetal heart rate monitoring
mobile applications. Telemed J E Health 2019; 25 (9): 870–7.
45. Furlong L, Morris M, Serry T, Erickson S. Mobile apps for treatment of
speech disorders in children: an evidence-based analysis of quality and ef-
ficacy. PLoS One 2018; 13 (8): e0201513.
46. Barnes KL, Dunivan G, Jaramillo-Huff A, Krantz T, Thompson J, Jep-
pson P. Evaluation of smartphone pelvic floor exercise applications using
standardized scoring system. Female Pelvic Med Reconstr Surg 2019; 25
(4): 328–35.
47. Daly LM, Boyle FM, Gibbons K, Le H, Roberts J, Flenady V. Mobile
applications providing guidance about decreased fetal movement: review
and content analysis. Women Birth 2019; 32 (3): e289–96.
48. Portelli P, Eldred C. A quality review of smartphone applications for the
management of pain. Br J Pain 2016; 10 (3): 135–40.
49. Guo Y, Bian J, Leavitt T, et al. Assessing the quality of mobile exercise
apps based on the American College of Sports Medicine Guidelines: a
reliable and valid scoring instrument. J Med Internet Res 2017; 19 (3):
e67.
50. Huckvale K, Adomaviciute S, Prieto JT, Leow MKS, Car J. Smartphone
apps for calculating insulin dose: a systematic assessment. BMC Med
2015; 13: 106.
51. Xiao Q, Lu S, Wang Y, Sun L, Wu Y. Current status of cardiovascular
disease-related smartphone apps downloadable in China. Telemed J E
Health 2017; 23 (3): 219–25.
52. Kochmann M, Locatis C. Direct to consumer mobile teledermatology
apps: an exploratory study. Telemed E Health 2016; 22 (8): 689–93.
53. Sullivan RK, Marsh S, Halvarsson J, et al. Smartphone apps for measur-
ing human health and climate change co-benefits: a comparison and
quality rating of available apps. JMIR mHealth Uhealth 2016; 4 (4):
e135.
54. Regmi K, Kossim N, Ahmod NH, Tuoh NA. Assessment of content,
quality and compliance of the STaR mobile application for smoking ces-
sation. Tob Prev Cessation 2017; 3: 120.
55. Flaherty S-J, McCarthy M, Collins A, McAuliffe F. Can existing mobile
apps support healthier food purchasing behaviour? Content analysis of
nutrition content, behaviour change theory and user quality integration.
Public Health Nutr 2018; 21 (2): 288–98.
56. Choi SK, Yelton B, Ezeanya VK, Kannaley K, Friedman DB. Review of
the content and quality of mobile applications about Alzheimer’s disease
and related dementias. J Appl Gerontol 2018 Jul 26 [E-pub ahead of
print].
57. Reynoldson C, Stones C, Allsop M, et al. Assessing the quality and us-
ability of smartphone apps for pain self-management. Pain Med 2014;
15 (6): 898–909.
58. Jeon E, Park HA, Min YH, Kim HY. Analysis of the information quality
of Korean obesity-management smartphone applications. Healthc In-
form Res 2014; 20 (1): 23–9.
59. Orr K, Howe HS, Omran J, et al. Validity of smartphone pedometer
applications. BMC Res Notes 2015; 8 (1): 733.
60. Bailey SC, Belter LT, Pandit AU, Carpenter DM, Carlos E, Wolf MS.
The availability, functionality, and quality of mobile applications sup-
porting medication selfmanagement. J Am Med Inform Assoc 2014; 21
(3): 542–6.
61. Lee H, Sullivan SJ, Schneiders AG, et al. Smartphone and tablet apps for
concussion road warriors (team clinicians): a systematic review for prac-
tical users. Br J Sports Med 2015; 49 (8): 499–505.
62. Gibbs J, Gkatzidou V, Tickle L, et al. ‘Can you recommend any good STI
apps?’ A review of content, accuracy and comprehensiveness of current
mobile medical applications for STIs and related genital infections. Sex
Transm Infect 2017; 93 (4): 234.1.
63. Kumar S, Mehrotra S. Free mobile apps on depression for Indian users: a
brief overview and critique. Asian J Psychiatr 2017; 28: 124–30.
64. Coppetti T, Brauchlin A, Mu¨ ggler S, et al. Accuracy of smartphone apps
for heart rate measurement. Eur J Prev Cardiol 2017; 24 (12): 1287–93.
65. Singh K, Drouin K, Newmark LP, et al. Many mobile health apps target
high-need, high-cost populations, but gaps remain. Health Aff (Mill-
wood) 2016; 35 (12): 2310–8.
66. Gray LJ, Leigh T, Davies MJ, et al. Systematic review of the develop-
ment, implementation and availability of smart-phone applications for
assessing type 2 diabetes risk. Diabet Med 2013; 30 (6): 758–60.
67. Baig MM, GholamHosseini H, Connolly MJ. Mobile healthcare applica-
tions: system design review, critical issues and challenges. Australas Phys
Eng Sci Med 2015; 38 (1): 23–38.
68. Thissen M, Udrea A, Hacking M, von Braunmuehl T, Ruzicka T.
mHealth app for risk assessment of pigmented and nonpigmented skin
lesions-a study on sensitivity and specificity in detecting malignancy. Tel-
emed J E Health 2017; 23 (12): 948–54.
69. Patel P, Kim JY, Brooks LJ. Accuracy of a smartphone application in esti-
mating sleep in children. Sleep Breath 2017; 21 (2): 505–11.
70. Penzenstadler L, Chatton A, Van Singer M, Khazaal Y. Quality of smart-
phone apps related to alcohol use disorder. Eur Addict Res 2016; 22 (6):
329–38.
71. Xiao Q, Wang YL, Sun L, Lu S, Wu Y. Current Status and quality assess-
ment of cardiovascular diseases related smartphone apps in China. Stud
Health Technol Inform 2016; 225: 1030–1; IOS press.
72. Park JYE, Li J, Howren A, Tsao NW, De Vera M. Mobile phone apps
targeting medication adherence: quality assessment and content analysis
of user reviews. JMIR mHealth Uhealth 2019; 7 (1): e11919.
73. Carter T, O’Neill S, Johns N, Brady RRW. Contemporary vascular
smartphone medical applications. Ann Vasc Surg 2013; 27 (6): 804–9.
74. Gajecki M, Berman AH, Sinadinovic K, Rosendahl I, Andersson C. Mo-
bile phone brief intervention applications for risky alcohol use among
university students: a randomized controlled study. Addict Sci Clin Pract
2014; 9: 11.
75. Mobasheri MH, Johnston M, King D, Leff D, Thiruchelvam P, Darzi A.
Smartphone breast applications - what’s the evidence? Breast 2014; 23
(5): 683–9.
Journal of the American Medical Informatics Association, 2019, Vol. 0, No. 0 9
Downloaded from https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocz175/5585394 by guest on 16 October 2019
76. Lee Y, Moon M. Utilization and content evaluation of mobile applica-
tions for pregnancy, birth, and child care. Healthc Inform Res 2016; 22
(2): 73–80.
77. Kwan V, Bihelek N, Anderson V, Yeates K. A review of
smartphone applications for persons with traumatic brain injury: what is
available and what is the evidence? J Head Trauma Rehabil 2019; 34 (2):
E45–51.
78. Sim~
oes P, Silva AG, Amaral J, Queir
os A, Rocha NP, Rodrigues M. Fea-
tures, behavioral change techniques, and quality of the most popular mo-
bile apps to measure physical activity: Systematic search in app stores.
JMIR mHealth Uhealth 2018; 6 (10): e11281.
79. Jamaladin H, van de Belt TH, Luijpers LC, et al. Mobile apps for blood
pressure monitoring: systematic search in app stores and content analy-
sis. JMIR mHealth Uhealth 2018; 6 (11): e187.
80. Paudel N. Smartphone applications for amblyopia treatment: a review of
current apps and professional involvement. Telemed J E Health 2018; 24
(10): 797–802.
81. Pereira-Azevedo N, Carrasquinho E, Cardoso de Oliveira E, et al.
mHealth in urology: a review of experts’ involvement in app develop-
ment. PLoS One 2015; 10 (5): e0125547.
82. Pandey A, Hasan S, Dubey D, Sarangi S. Smartphone apps as a source of
cancer information: changing trends in health information-seeking be-
havior. J Cancer Educ 2013; 28 (1): 138–42.
83. Zhou AH, Patel VR, Baredes S, Eloy JA, Hsueh WD. Mobile
applications for allergic rhinitis. Ann Otol Rhinol Laryngol 2018; 127
(11): 836–40.
84. Casale M, Costantino A, Rinaldi V, et al. Mobile applications in otolar-
yngology for patients: an update. Laryngoscope Investig Otolaryngol
2018; 3 (6): 434–8.
85. Regmi D, Tobutt C, Shaban S. Quality and use of free smoking cessation
apps for smartphones. Int J Technol Assess Health Care 2018; 34 (5):
476–80.
86. Zhang MW, Ho RC, Hawa R, Sockalingam S. Analysis of the informa-
tion quality of bariatric surgery smartphone applications using the Sil-
berg scale. Obes Surg 2016; 26 (1): 163–8.
87. Van Singer M, Chatton A, Khazaal Y. Quality of smartphone apps re-
lated to panic disorder. Front Psychiatry 2015; 6: 96.
88. Fougerouse PA, Yasini M, Marchand G, Aalami OO. A cross-sectional
study of prominent US mobile health applications: evaluating the current
landscape. AMIA Annu Symp Proc 2017; 2017: 715–23.
89. Zhang MWB, Ho RCM, Loh A, et al. Current status of postnatal depres-
sion smartphone applications available on application stores: an infor-
mation quality analysis. BMJ Open 2017; 7 (11): e015655.
90. Owens OL, Beer JM, Reyes LI, Thomas TL. Systematic review of com-
mercially available mobile phone applications for prostate cancer educa-
tion. Am J Mens Health 2018; 13 (1): 1557988318816912.
91. Richardson B, Dol J, Rutledge K, et al. Evaluation of mobile apps tar-
geted to parents of infants in the neonatal intensive care unit: systematic
app review. JMIR mHealth Uhealth 2019; 7 (4): e11620.
92. Wong SJ, Robertson GA, Connor KL, Brady RR, Wood AM. Smart-
phone apps for orthopaedic sports medicine - a smart move? BMC Sports
Sci Med Rehabil 2015; 7: 23.
93. Lee J. Hype or hope for diabetes mobile health applications? Diabetes
Res Clin Pract 2014; 106 (2): 390–2.
94. Hundert AS, Huguet A, McGrath PJ, Stinson JN, Wheaton M. Commer-
cially available mobile phone headache diary apps: a systematic review.
JMIR Mhealth Uhealth 2014; 16 (8): e36.
95. Fraser H, Coiera E, Wong D. Safety of patient-facing digital symptom
checkers. Lancet 2018; 392 (10161): 2263–4.
96. Chen Q, Carbone ET. Functionality, implementation, impact, and the
role of health literacy in mobile phone apps for gestational diabetes:
scoping review. JMIR Diabetes 2017; 2 (2): e25.
97. Huckvale K, Prieto JT, Tilney M, Benghozi PJ, Car J. Unaddressed pri-
vacy risks in accredited health and wellness apps: a cross-sectional sys-
tematic assessment. BMC Med 2015; 13 (1): 214.
98. Schnall R, Rojas M, Bakken S, et al. A user-centered model for designing
consumer mobile health (mHealth) applications (apps). J Biomed Inform
2016; 60: 243–51.
99. Stanziola E, Uznayo MQ, Ortiz JM, et al. User-centered design of health
care software development: towards a cultural change. Stud Health
Technol Inform 2015; 216: 368–71.
100. O’Neill S, Brady RR. Colorectal smartphone apps: opportunities and
risks. Colorectal Dis 2012; 14 (9): e530–4.
101. Rosser BA, Eccleston C. Smartphone applications for pain management.
J Telemed Telecare 2011; 17 (6): 308–12.
102. Coiera E. Guide to Health Informatics. 3 ed: CRC Press; 2015.
103. Curtis K, Lebedev A, Aguirre E, Lobitz S. A medication adherence app
for children with sickle cell disease: qualitative study. JMIR mHealth
Uhealth 2019; 7 (6): e8130.
104. Demidowich AP, Lu K, Tamler R, Bloomgarden Z. An evaluation of dia-
betes self-management applications for android smartphones. J Telemed
Telecare 2012; 18 (4): 235–8.
105. Yasini M, Marchand G. Mobile health applications, in the absence of
an authentic regulation. Stud Health Technol Inform 2015; 216: 127–31.
106. Corrao NJ, Robinson AG, Swiernik MA, Naeim A. Importance of testing
for usability when selecting and implementing an electronic health or
medical record system. J Oncol Pract 2010; 6 (3): 120–4.
107. Berkowitz CM, Zullig LL, Koontz BF, Smith SK. Prescribing an app?
Oncology providers’ views on mobile health apps for cancer care. JCO
Clin Cancer Inform 2017; 1: 1–7.
108. Kessel KA, Vogel MME, Schmidt-Graf F, Combs SE. Mobile apps in on-
cology: a survey on health care professionals’ attitude toward telemedi-
cine, mHealth, and oncological apps. J Med Internet Res 2016; 18 (11):
e312.
109. Byambasuren O, Beller E, Glasziou P. Current knowledge and adoption
of mobile health apps among Australian general practitioners: survey
study. JMIR mHealth Uhealth 2019; 7 (6): e13199.
110. Roper L, Jorm C. Please leave your phone outside: policymakers and
medical app providers are encouraging patient participation, but doctors
in the consult room are not. BMJ Innov 2017; 3 (2): 65–70.
111. Kessel KA, Vogel MM, Kessel C, et al. Mobile health in oncology: a pa-
tient survey about app-assisted cancer care. JMIR mHealth Uhealth
2017; 5 (6): e81.
112. Boudreaux ED, Waring ME, Hayes RB, Sadasivam RS, Mullen S, Pagoto
S. Evaluating and selecting mobile health apps: strategies for healthcare
providers and healthcare organizations. Behav Med Pract Policy Res
2014; 4 (4): 363–71.
113. Sharwood S. TGA to update regulation for health apps. itNews 2018.
https://www.itnews.com.au/news/tga-to-update-regulation-for-health-
apps-513805. Accessed November 23, 2018.
114. National Health Information Board. Workstream 1.2 Roadmap: Enable
Me to Make the Right Health and Care Choices. 2015. https://assets.
publishing.service.gov.uk/government/uploads/system/uploads/attach-
ment_data/file/442833/Work_Stream_1_2.pdf. Accessed November 23,
2018.
115. Ministry of Health NZ. Guidance on evaluating or developing a health
app. 2017. https://www.health.govt.nz/system/files/documents/pages/gu
idance-evaluating-developing-health-app-oct17-v2.pdf. Accessed November
23, 2018.
116. van Velthoven M, Powell J. Do health apps need endorsement? Chal-
lenges for giving advice about which health apps are safe and effective to
use. Digit Health 2017; 3: 205520761770134.
117. U.S. Food and Drug Administration. Digital Health Software Precertifi-
cation (Pre-Cert) Program 2018. https://www.fda.gov/medical-devices/
digital-health/digital-health-software-precertification-pre-cert-program#
targetText¼The%20Software%20Precertification%20(Pre%2DCert,-
they%20reach%20the%20U.S.%20market. Accessed November 23,
2018.
118. Consumers Health Forum of Australia. Results of Australia’s Health
Panel survey on recommendations and regulation of smartphone apps
10 Journal of the American Medical Informatics Association, 2019, Vol. 0, No. 0
Downloaded from https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocz175/5585394 by guest on 16 October 2019
for health and wellness. 2018. https://www.chf.org.au/sites/default/files/
results_of_australias_health_panel_survey_on_recommendations_and_
regulation_of_smartphone_apps_for_health_and_wellness.pdf. Accessed
November 23, 2018.
119. Hill S. Can you really trust app store ratings? We asked the experts. Digi-
tal Trends. 2018. https://www.digitaltrends.com/android/can-you-re-
ally-trust-app-store-ratings/. Accessed November 23, 2018.
120. NHS Apps Library: National Health Service. https://apps.beta.nhs.uk/.
Accessed December 7, 2018.
121. Bates DW, Landman A, Levine DM. Health apps and health policy:
what is needed? JAMA 2018; 320 (19): 1975–6.
122. El-Sayed H, Chaqfeh M. Exploiting mobile edge computing for enhanc-
ing vehicular applications in smart cities. Sensors (Basel) 2019; 19 (5):
E1073.
123. Eysenbach G. CONSORT-EHEALTH: implementation of a checklist for
authors and editors to improve reporting of web-based and mobile
randomized controlled trials. Stud Health Technol Inform 2013; 192:
657–61.
124. Brender J, Talmon J, Keizer ND, Nyk€
anen P, Rigby M, Ammenwerth E.
STARE-HI - statement on reporting of evaluation studies in health infor-
matics: explanation and elaboration. Appl Clin Inform 2013; 4 (3):
331–58.
Journal of the American Medical Informatics Association, 2019, Vol. 0, No. 0 11
Downloaded from https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocz175/5585394 by guest on 16 October 2019
