Scaling Up mHealth: Where Is the Evidence?
Mark Tomlinson1*, Mary Jane Rotheram-Borus2, Leslie Swartz3, Alexander C. Tsai4,5
1Centre for Public Mental Health, Department of Psychology, Stellenbosch University, Stellenbosch, South Africa, 2Semel Institute for Neuroscience and Human
Behaviour, University of California at Los Angeles, Los Angeles, California, United States of America, 3Centre for Public Mental Health, Department of Psychology,
Stellenbosch University, Stellenbosch, South Africa, 4Chester M. Pierce, MD Division of Global Psychiatry, Department of Psychiatry, Massachusetts General Hospital,
Boston, Massachusetts, United States of America, 5Center for Global Health, Massachusetts General Hospital, Boston, Massachusetts, United States of America
What Is the Problem?
There are over 6 billion mobile phone
subscribers and 75% of the world has
access to a mobile phone . Service and
care providers, researchers, and national
governments are excited at the opportu-
nities mobile health has to offer in terms of
improving access to health care, engage-
ment and delivery, and health outcomes
. Interventions categorized under the
rubric ‘‘mobile health’’ or ‘‘mHealth’’—
broadly defined as medical and public
health practice supported by mobile de-
vices —span a variety of applications
ranging from the use of mobile phones to
improve point of service data collection
, care delivery , and patient com-
munication  to the use of alternative
wireless devices for real-time medication
monitoring and adherence support .
A recent World Bank report tracked
more than 500 mHealth studies, and
many donor agencies are lining up to
support the ‘‘scaling up’’ of mHealth
interventions . Yet, after completion of
these 500 pilot studies, we know almost
nothing about the likely uptake, best
strategies for engagement, efficacy, or
effectiveness of these initiatives. Currently,
mHealth interventions lack a foundation
of basic evidence , let alone a founda-
tion that would permit evidence-based
scale up. For example, in Uganda in
2008 and 2009 approximately 23 of 36
mHealth initiatives did not move beyond
the pilot phase . The current enthusi-
asm notwithstanding, the scatter-shot ap-
proach to piloting mHealth projects in the
absence of a concomitant programmatic
implementation and evaluation strategy
may dampen opportunities to truly capi-
talize on the technology. This article
discusses a number of points pertinent to
developing a more robust evidence base
for the scale up of mHealth interventions.
The issues raised are primarily conceptual
Industry’s increasing role in pushing for
mHealth scale up is also a cause for
concern. At a recent mHealth conference
in South Africa, there were repeated calls
for scale up of mHealth initiatives across
low- and middle-income countries (LA-
MICs). Many of these calls emanated from
industry representatives rather than re-
searchers, governments, or care providers
. It is likely that private enterprise has
a quite different understanding of what
scale up means, with growing market
share, rather than improved health out-
comes, at the core of their mission. The
growing involvement by industry, predom-
inantly mobile phone providers, warrants
some caution in addition to perhaps a
code of practice. Public–private partner-
ships will be of central importance in the
evolution of the mHealth field (as we
discuss later), but this cannot happen at
the expense of good science and good
In some ways, mobile technology has a
magical appeal for those interested in
global public health over and above the
advantages that have been proven with
good evidence . Part of this magical
promise is that mobile technologies may
solve one of the most difficult problems
facing global health efforts—that of struc-
tural barriers to access. Travel, especially
to remote areas in LAMICs, is expensive,
destructive to the environment, time-
consuming, and exhausting and physically
challenging to many. In the global health
field, there are many practitioners whose
personal and working lives are substan-
tially disrupted by travel of this nature.
Mobile technology may hold out the
promise of a world within which these
difficulties can be minimised or eliminat-
ed. There is an obvious appeal for people
from higher-income contexts being able to
remain at home and in their offices while
interacting with and improving the health
of people very far away and in straitened
circumstances. Mobile technology may
hold out the promise that the visceral
challenges of travel and complex intercul-
tural contact, so much a feature of the
global health enterprise, may now be a
thing of the past .
Current State of the Evidence
While enthusiasm for effective mHealth
interventions in sub-Saharan Africa is
high, little is known about their efficacy
or effectiveness. Most randomized trials of
mHealth interventions have employed text
message reminder systems. Two systemat-
ic reviews have described a robust evi-
dence base for the use of text message
reminders to improve attendance at health
care appointments [13,14]. Yet, none of
the studies included in these reviews was
conducted in resource-limited settings.
The Essay section contains opinion pieces on topics
of broad interest to a general medical audience.
Citation: Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC (2013) Scaling Up mHealth: Where Is the
Evidence? PLoS Med 10(2): e1001382. doi:10.1371/journal.pmed.1001382
Published February 12, 2013
Copyright: ? 2013 Tomlinson et al. This is an open-access article distributed under the terms of the Creative
Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium,
provided the original author and source are credited.
Funding: No specific funding was received for writing this article. MT acknowledges the support of the
National Research Foundation (South Africa) and the Department for International Development (DfID-UK). ACT
acknowledges salary support from U.S. National Institutes of Health K23 MH-096620.
Competing Interests: MT is a member of the PLOS Medicine Editorial Board. AT receives salary support from
NIH K23 MH-096620.
Abbreviations: EBI, evidence-based intervention; LAMIC, low- and middle-income country; mHealth, mobile
health; MOST, Multiphase Optimization Strategy.
* E-mail: firstname.lastname@example.org
Provenance: Not commissioned; externally peer reviewed.
PLOS Medicine | www.plosmedicine.org1 February 2013 | Volume 10 | Issue 2 | e1001382
Similarly, few randomized trials evaluating
the use of text message reminders to
improve medication adherence for people
with chronic illnesses have been conducted
in LAMICs [15–18]. Three randomized
trials studying HIV treatment adherence
found benefits [19,20] and one found no
impact [21,22]. Two recent systematic
reviews [23,24] found modest and sugges-
tive evidence for the benefits of mHealth
technology, and while both reviews rec-
ommended implementation, they argued
that high quality (and adequately powered)
clinical trials that measure clinical out-
comes are essential.
The reviews of mHealth interventions
would be more helpful if the results were
organized according to 1) foundational
functions (informing, training, monitoring,
shaping, supporting, and linking to care);
2) content-specific targets (e.g., for Millen-
nium Development Goal developmentally
related tasks and challenges); and 3) local
cultural adaptations (e.g., language) .
The inconsistency of results from mHealth
studies demonstrates the importance of
having an organizational framework.
What Constitutes Evidence?
The Institute of Medicine  and
other communities of researchers [27,28]
have established standards for the phases
of research that must be conducted in
order to be considered efficacious, effec-
tive, and disseminated. Flay and col-
leagues  have adapted the evidentiary
standards model published by the Society
for Prevention Research . These
standards were developed in order to
guide policy, research, and practice and
provide a useful framework to determine
what constitutes good and sufficient evi-
dence. In this model (see Figure 1), scale
up or country-wide implementation would
be dependent on the completion (for each
intervention) of (a) two high quality
efficacy trials, (b) two high quality effec-
tiveness trials, followed by (c) dissemina-
tion research that has established that the
intervention can be delivered with fidelity
to the model being tested, as well as (d)
information about the intervention’s costs.
There are currently no mHealth interven-
tions that meet these standards for scale up
Linked to the issue of standards for the
of behaviour change. Aboud and Singla
 have shown how programmes that
simply provide health information (e.g., via
SMS [short message service, or text
messaging]) tend to be unsuccessful, while
interventions providing skills through peer
educators are more likely to be successful
. There are well validated theories of
behaviour change common to many evi-
dence-based interventions for prevention,
diagnoses, and care, but none of the
mhealth initiatives appear to be grounded
in such theories . We would argue that
in the context of scarce resources, imple-
menting untested mHealth interventions at
scale without a theory of behaviour change
is likely to result in many failed scale up
projects and significant levels of wasted
Finally, no major investments have been
made to create a robust platform for
mobile phones that could be used by
designers of applications and electronic
medical records that will allow cross-
fertilization or integrated systems to be
utilized . Thousands of small applica-
tions have been propagated on closed-
source platforms (e.g., iPhone applications
and others) that each major mobile phone
provider appears ready to replicate at high
cost. Currently, a patient with two or more
health conditions will have to make use of
numerous applications for monitoring
different health-related parameters such
status, a disease-specific approach that he
or she is unlikely to sustain . Estrin
and Sim make the case that there is a
global communication network already in
place to support an open mHealth archi-
tecture that could facilitate scalable and
sustainable health information systems
. Interoperability will be critical to
promote research initiatives. The largest
investments to date in interoperable sys-
tems have been actively pursued by for-
profit companies, given the staggering
profits to be made in the proprietary
applications market. What is needed is a
concerted effort by governments, funders,
and private enterprise to cooperate in
order to set standards (e.g., number of bits)
and to create a self-governing commer-
cially viable ecosystem for innovation .
mHealth is in a period very similar to the
early days of the Internet: not creating
robust, interoperable platforms will ensure
failure for mHealth initiatives to be scaled
to improve health outcomes for at least the
What Needs to Happen Next:
From Black Box to High Utility
The current wave of mHealth interven-
tions are the equivalent of black boxes.
Each small entrepreneur or researcher
includes whatever bells and whistles that
their funding allows in an attempt to
demonstrate efficacy. For example, hun-
dreds of small pilot studies are finding
whether text messaging works. Text mes-
saging is more likely to work under a set of
N when there is follow-up;
N when the message is personally tai-
N when the frequency, wording, and
content are highly relevant.
Similar strategies are being experiment-
ed with for a range of topics, delivery
strategies (web, phone, videos, social
media sties), and populations. There are
a set of principles that could potentially be
established to identify the optimal strate-
gies for delivering mHealth interventions.
However, our current research is not
aimed at identifying these principles and
strategies. Each pilot study is examining
whether their particular style of a black
box application works better than not
having any black box application. It is
time to start funding randomized con-
trolled trials of interventions that are based
on researchers’ best guesses about optimal
It is also time to consider the Multi-
phase Optimization Strategy (MOST)
developed by Collins and colleagues .
The MOST strategy is grounded in an
engineering approach and requires a two-
stage process: 1) identifying the range of
features that contribute to variation for a
particular intervention; and 2) selecting a
small set of factors and empirically testing
them with a multi-factorial design. The
N Despite hundreds of mHealth pilot studies, there has been insufficient
programmatic evidence to inform implementation and scale-up of mHealth.
N We discuss what constitutes appropriate research evidence to inform scale up.
N Potential innovative research designs such as multi-factorial strategies,
randomized controlled trials, and data farming may provide this evidence base.
N We make a number of recommendations about evidence, interoperability, and
the role of governments, private enterprise, and researchers in relation to the
scale up of mHealth.
PLOS Medicine | www.plosmedicine.org2February 2013 | Volume 10 | Issue 2 | e1001382
initial set of factors to be screened might
be determined on the basis of theory and/
or experience and could be informed by
research implementing evidence-based in-
terventions with other delivery formats.
The utility of such an approach has been
demonstrated by Stretcher and colleagues
 for web-based smoking cessation
policies. Rather than having a single tested
web-based, evidence-based intervention
(EBI) that will then compete with other
web-based EBI for smoking, there are a set
of parameters that outline the optimal
strategy for implementing a web-based
programme. Similar strategies have long
been adopted by health services research-
ers . However, few of the existing
studies utilizing mHealth delivery formats
have adopted such an approach.
MOST is not the only approach that
could potentially enhance the efficiency of
existing mHealth studies. Duan  has
advocated for the establishment of data
farms. Nascent Internet companies such as
Google, Yahoo, and Facebook provide
informative case studies of data farms.
Rather than use experimental research
designs (such as randomized controlled
trials), these companies can harvest data
from billions of users of mobile, web, and
social media, and computer-based inter-
ventions provide the evidence regarding
the specific types of consumers who are
attracted to specific types of delivery
formats delivered with specific levels of
doses at specific times. Data farms offer
the opportunity to know the who, what,
when, where, and how of reaching con-
sumers . Private enterprise has been
outstanding at this function: mHealth
needs to utilize their platforms and
methods to optimize personal health.
Major donors could invest in creating a
robust set of standards and a platform that
can inform and support local adaptation of
mHealth applications. The standardized
features of the platform could then be
available to all local technicians commit-
ted to improving the health of their local
communities. At the very least, given that
standards are expensive to establish, as
Figure 1. Research stages and standards. Adapted from Olds et al.  and Flay et al. .
PLOS Medicine | www.plosmedicine.org3February 2013 | Volume 10 | Issue 2 | e1001382
well as often being complex and difficult to
understand, one option is for an organi-
zation such as the World Health Organi-
zation to ‘‘certify’’ standards that meet
particular criteria, or even to become a
disseminator of standards. We also believe
a global strategy for programmatic exam-
ination of the optimal features of the
mobile platforms is needed, namely a
platform that incorporates (for example)
factorial designs to test the multiple
features of interventions , the MOST
strategy and even data farms. This could
quickly identify and provide guidance to
hundreds of thousands of programmers
globally that could leverage donor invest-
ments to improve their communities’
access to information, skills, telemedicine,
or management of front line workers.
Wrote the first draft of the manuscript: MT
MR-B. Contributed to the writing of the
manuscript: MT AT LS MR-B. ICMJE criteria
for authorship read and met: MT AT LS MR-
B. Agree with manuscript results and conclu-
sions: MT AT LS MR-B.
1. ICD (2012) Information and communications for
development: maximizing mobile. Available:
cessed 6 January 2013.
2. van Heerden A, Tomlinson M, Swartz L (2012)
Point of care in your pocket: a research agenda
for the field of m-health. Bull World Health
Organ 90: 393–394.
3. Tomlinson M, Solomon W, Singh Y, Doherty T,
Chopra M, et al. (2009) The use of mobile phones
as a data collection tool: a report from a
household survey in South Africa. BMC Med
Inform Decis Mak 9: 51.
4. Rotheram-Borus MJ, le Roux IM, Tomlinson M,
Mbewu N, Comulada WS, et al. (2011) Philani
Plus (+): a Mentor Mother community health
worker home visiting program to improve mater-
nal and infants’ outcomes. Prev Sci 12: 372–388.
5. Siedner MJ, Haberer JE, Bwana MB, Ware NC,
Bangsberg DR (2012) High acceptability for cell
phone text messages to improve communication
of laboratory results with HIV-infected patients in
rural Uganda: a cross-sectional survey study.
BMC Med Inform Decis Mak 12: 56.
6. Haberer JE, Robbins GK, Ybarra M, Monk A,
Ragland K, et al. (2012) Real-time electronic
adherence monitoring is feasible, comparable to
unannounced pill counts, and acceptable. AIDS
Behav 16: 375–382.
7. Qiang CZ, Yamamichi M, Hausman V, Altman
D (2011) Mobile applications for the health
sector. Washington: World Bank.
8. Collins F (2012) How to fulfill the true promise of
‘‘mHealth’’: Mobile devices have the potential to
become powerful medical tools. Sci Am 307: 16.
9. Lemaire J (2011) Scaling up mobile health:
Elements necessary for the successful scale up of
mHealth in developing countries. Geneva: Ad-
vanced Development for Africa.
10. Friedrichs C (2012) Market access & reimburse-
ment strategies. Mobile Health Summit. Cape
Town. 29 May–1 June 2012.
11. Piette JD, Lun KC, Moura LA, Jr., Fraser HS,
Mechael PN, et al. (2012) Impacts of e-health on
the outcomes of care in low- and middle-income
countries: where do we go from here? Bull World
Health Organ 90: 365–372.
12. Turkle S (2012) Alone together: why we expect
more of technology and less of each other. New
York: Basic Books.
13. Car J, Gurol-Urganci I, de Jongh T, Vodopivec-
Jamsek V, Atun R (2012) Mobile phone messag-
ing reminders for attendance at healthcare
appointments. Cochrane Database Syst Rev 7:
14. Guy R, Hocking J, Wand H, Stott S, Ali H, et al.
(2012) How effective are short message service
reminders at increasing clinic attendance? A
meta-analysis and systematic review. Health Serv
Res 47: 614–632.
15. Vervloet M, Linn AJ, van Weert JC, de Bakker
DH, Bouvy ML, et al. (2012) The effectiveness of
interventions using electronic reminders to im-
prove adherence to chronic medication: a sys-
tematic review of the literature. J Am Med Inform
Assoc 19: 696–704.
16. Krishna S, Boren SA, Balas EA (2009) Healthcare
via cell phones: a systematic review.
Telemed J E Health 15: 231–240.
17. Horvath T, Azman H, Kennedy GE, Rutherford
GW (2012) Mobile phone text messaging for
promoting adherence to antiretroviral therapy in
patients with HIV infection. Cochrane Database
Syst Rev 3: CD009756.
18. Cole-Lewis H, Kershaw T (2010) Text messaging
as a tool for behavior change in disease prevention
and management. Epidemiol Rev 32: 56–69.
19. Lester RT, Ritvo P, Mills EJ, Kariri A, Karanja
S, et al. (2010) Effects of a mobile phone short
message service on antiretroviral treatment ad-
herence in Kenya (WelTel Kenya1): a rando-
mised trial. Lancet 376: 1838–1845.
20. Pop-Eleches C, Thirumurthy H, Habyarimana
JP, Zivin JG, Goldstein MP, et al. (2011) Mobile
phone technologies improve adherence to anti-
retroviral treatment in a resource-limited setting:
a randomized controlled trial of text message
reminders. AIDS 25: 825–834.
21. da Costa TM, Barbosa BJ, Gomes e Costa DA,
Sigulem D, de Fatima Marin H, et al. (2012)
Results of a randomized controlled trial to assess
the effects of a mobile SMS-based intervention on
treatment adherence in HIV/AIDS-infected Bra-
zilian women and impressions and satisfaction
with respect to incoming messages. Int J Med
Inform 81: 257–269.
22. Zurovac D, Sudoi RK, Akhwale WS, Ndiritu M,
Hamer DH, et al. (2011) The effect of mobile phone
text-message reminders on Kenyan health workers’
adherence to malaria treatment guidelines: a cluster
randomised trial. Lancet 378: 795–803.
23. Free C, Phillips G, Galli L, Watson L, Felix L, et
al. (2013) The effectiveness of mobile-health
technology based health behaviour change or
disease management interventions for health care
consumers: a systematic review PLoS Med 10:
e1001362. doi:10.1371/journal. pmed.1001362
24. Free C, Phillips G, Watson L, Galli L, Felix L, et
al. (2013) The effectiveness of M-health technol-
ogies to improve health care service delivery
processes: a systematic review and meta-analysis.
PLoS Med 10: e1001363. doi:10.1371/journal.
25. Rotheram-Borus MJ, Tomlinson M, Swendeman
D, Lee A, Jones E (2012) Standardized functions
for smart-phone applications: examples from
maternal and child health. International Journal
of Telemedicine and Applications 2012.
26. Mrazek PJ, Haggerty RJ (1994) Reducing risks for
mental disorders: Frontiers for preventive inter-
vention research. Washington, D.C.: Committee
on Prevention of Mental Disorders: Institute of
27. Flay BR, Biglan A, Boruch RF, Castro FG,
Gottfredson D, et al. (2005) Standards of
evidence: criteria for efficacy, effectiveness and
dissemination. Prev Sci 6: 151–175.
28. Thornicroft G, Lempp H, Tansella M (2011) The
place of implementation science in the transla-
tional medicine continuum. Psychol Med 41:
29. Olds DL, Sadler L, Kitzman H (2007) Programs
for parents of infants and toddlers: recent
evidence from randomized trials. J Child Psychol
Psychiatry 48: 355–391.
30. Aboud FE, Singla DR (2012) Challenges to
changing health behaviours in developing coun-
tries: a critical overview. Soc Sci Med 75: 589–
31. Fishbein M, Bandura A, Triandis HC, Kanfer
FH, Becker M, et al. (2000) Factors influencing
behavior and behavior change. In: Baum AS,
Revenson T, Singer J, editors. Handbook of
Health Psychology. New Jersey: Lawrence Ear-
32. Estrin D, Sim I (2010) Health care delivery. Open
mHealth architecture: an engine for health care
innovation. Science 330: 759–760.
Box 1. Recommendations for Scale Up of mHealth
1. Existing standards for research should be reconsidered in order to provide
guidance as to when scale up is appropriate.
2. mHealth interventions should be guided by a plausible theory of behaviour
change and should use more than one technique depending on the targeted
3. We need to establish an open mHealth architecture based on a robust platform
with standards for app development which would facilitate scalable and
sustainable health information systems.
4. Implementation strategies such as factorial designs that are able to test the
multiple features of interventions must be explored, in order to provide the
necessary evidence base.
5. Scale-up of mHealth in LAMICs should be preceded by efficacy and effectiveness
trials so that they are founded on an appropriate evidence base.
6. Governments, funders, and industry must cooperate in order to set standards to
create a self-governing commercially viable ecosystem for innovation.
PLOS Medicine | www.plosmedicine.org4 February 2013 | Volume 10 | Issue 2 | e1001382
33. Collins LM, Baker TB, Mermelstein RJ, Piper Download full-text
ME, Jorenby DE, et al. (2011) The multiphase
optimization strategy for engineering effective
tobacco use interventions. Ann Behav Med 41:
34. Strecher VJ, McClure JB, Alexander GL, Chak-
raborty B, Nair VN, et al. (2008) Web-based
smoking-cessation programs: results of a random-
ized trial. Am J Prev Med 34: 373–381.
35. Duan N, Alegria M, Canino G, McGuire TG,
Takeuchi D (2007) Survey conditioning in self-
reported mental health service use: randomized
comparison of alternative instrument formats.
Health Serv Res 42: 890–907.
36. Hay MC, Weisner TS, Subramanian S, Duan N,
Niedzinski EJ, et al. (2008) Harnessing experience:
exploring the gap between evidence-based medicine
and clinical practice. J Eval Clin Pract 14: 707–713.
37. Collins LM, Chakraborty B, Murphy SA, Stre-
cher V (2009) Comparison of a phased experi-
mental approach and a single randomized clinical
trial for developing multicomponent behavioral
interventions. Clin Trials 6: 5–15.
38. Briscoe C, Aboud F (2012) Behaviour change
communication targeting four health behaviours
in developing countries: a review of change
techniques. Soc Sci Med 75: 612–621.
PLOS Medicine | www.plosmedicine.org5February 2013 | Volume 10 | Issue 2 | e1001382