Systematic review and evaluation of web-accessible
tools for management of diabetes and related
cardiovascular risk factors by patients and
Catherine H Yu,1,2Robinder Bahniwal,1Andreas Laupacis,1,2Eman Leung,1
Michael S Orr,3Sharon E Straus1,2
Objective To identify and evaluate the effectiveness,
clinical usefulness, sustainability, and usability of
web-compatible diabetes-related tools.
Data sources Medline, EMBASE, CINAHL, Cochrane
Central Register of Controlled Trials, world wide web.
Study selection Studies were included if they described
an electronic audiovisual tool used as a means to
educate patients, care givers, or clinicians about
diabetes management and assessed a psychological,
behavioral, or clinical outcome.
Data extraction Study abstraction and evaluation for
clinical usefulness, sustainability, and usability were
performed by two independent reviewers.
Results Of 12616 citations and 1541 full-text articles
reviewed, 57 studies met inclusion criteria. Forty studies
used experimental designs (25 randomized controlled
trials, one controlled clinical trial, 14 beforeeafter
studies), and 17 used observational designs.
Methodological quality and ratings for clinical usefulness
and sustainability were variable, and there was a high
prevalence of usability errors. Tools showed moderate
but inconsistent effects on a variety of psychological and
clinical outcomes including HbA1c and weight. Meta-
regression of adequately reported studies (12 studies,
2731 participants) demonstrated that, although the
interventions studied resulted in positive outcomes, this
was not moderated by clinical usefulness nor usability.
Limitation This review is limited by the number of
accessible tools, exclusion of tools for mobile devices,
study quality, and the use of non-validated scales.
Conclusion Few tools were identified that met our
criteria for effectiveness, usefulness, sustainability, and
usability. Priority areas include identifying strategies to
minimize website attrition and enabling patients and
clinicians to make informed decisions about website
choice by encouraging reporting of website quality
worldwide and is a leading cause of death in most
common in diabetes care. For example, in an
American population-based survey, only 62% of
patients with diabetes had low-density lipoprotein
cholesterol measured annually.2Reviews of inter-
providers to optimize diabetes care have shown
small effects on provider performance and patient
mellitus affects 285 million people
Clinical care gaps are
Given that consumers are increasingly using the
world wide web as a source of health information,6
web-based tools offer potential for optimizing
quality of diabetes care. Use of web-based media
may improve knowledge, social support, behavior
change, and clinical outcomes.7 8However, existing
diabetes websites have wide variations in the
quality of evidence provided9and offer didactic
information at high reading levels with little
interactive technology, social support, or problem-
solving assistance.10Similarly, although healthcare
providers increasingly use online resources for
patient care, the volume, breadth, editorial quality,
and evidence-based methodology upon which they
were developed are highly variable.11
The effectiveness of these tools in changing
clinical outcomes has been the subject of reviews in
other topic areas; for example, a systematic review
of consumer health informatics applications in
diverse topic areas, including breast cancer, found
outcomes.12Their effectiveness in a research setting
may not translate to effectiveness in clinical prac-
tice; factors that affect their adoption into clinical
practice include clinical usefulness, usability, and
sustainability.13Specifically, a clinically useful tool,
defined as a tool that provides clinically useful
answers and is easy to use, access, and read,14may
differ in a research context; for example, while
a website on carbohydrate counting may be useful
in a research setting with a research dietician, it
may be less useful to the consumer trying to use it
alone in a real-life setting. Similarly, usability of the
tool (defined as the extent to which a product can
be used by specified users to complete tasks
successfully, in time, and with satisfaction in
a specified context15) may be underemphasized in
research studies,16where participants are routinely
oriented to and trained on the use of the tool.
Finally, sustainability, defined as the degree to
which an innovation continues to be used after
initial effort to secure adoption is completed,17is
a critical component in addressing the gap between
research and practice,18yet is often not addressed or
assessed.19Critical appraisal of web tools should
thus consider their effectiveness and their clinical
usefulness, usability, and sustainability. Previous
studies have not evaluated the validity, clinical
usefulness, usability, and sustainability of web-
compatible, diabetes-related tools for patients and
providers, which was the objective of this study.
<Additional appendices are
published online only. To view
these files please visit the
journal online (www.jamia.bmj.
1Keenan Research Centre, Li Ka
Shing Knowledge Institute of
St Michael’s Hospital, Toronto,
2Department of Medicine,
Faculty of Medicine, University
of Toronto, Toronto, ON, Canada
3St Michael’s Hospital, Toronto,
Dr Catherine H Yu Keenan
Research Centre, Li Ka Shing
Knowledge Institute of St
Michael’s Hospital, 30 Bond St,
Toronto, ON M5B 1W8, Canada;
Received 9 April 2011
Accepted 27 November 2011
Published Online First
3 January 2012
This paper is freely available
online under the BMJ Journals
unlocked scheme, see http://
514 J Am Med Inform Assoc 2012;19:514e522. doi:10.1136/amiajnl-2011-000307
Because of the importance of multifactorial vascular risk
reduction as well as comprehensive lifestyle modification in the
care of patients with diabetes,20we were interested in diabetes-
specific tools and tools for blood pressure, lipid, smoking,
obesity, nutrition, physical activity and weight management.20
Data sources and searches
Published literature search strategy
EMBASE, CINAHL, and the Cochrane Central Register of
Controlled Trials were searched from their earliest date to June 1,
2011. The following search terms were used: diabetes, hyper-
tension, smoking cessation, weight reduction, online, computer-
based, and internet. The complete search strategy is provided in
online appendices. Additional articles were identified through
review of reference lists of identified studies and discussions
Grey literature search strategy
The world wide web was searched using the Google search
engine on June 14, 2009 with preselected phrases (online
appendices). We used these phrases to search websites of interest
that had been identified on the basis of expert knowledge.21
Sixty web ‘hits’ were captured for every phrase. The first 30 hits
that met our definition of a web-compatible diabetes care tool
Study inclusion and exclusion criteria
Studies were included if they evaluated a web-compatible
diabetes-related care tool, defined as an audiovisual tool that is
provided in electronic form to be used as a means to educate,
support, or advise patients, care givers, or healthcare providers,
and that addressed one of the following aspects of diabetes and
pre-diabetes management: glycemic control; cardiovascular risk
factor assessment and modification (including hypertension,
dyslipidaemia, and smoking cessation); nutrition, physical
psychological issues; and complication screening and manage-
ment. Tools were included if they addressed a relevant topic area,
regardless of whether patients with diabetes were the target
population. Studies could have observational or experimental
designs and had to include at least one psychological, behavioral,
or clinical outcome.
Studies were excluded if they (1) did not include an evaluation
of the tool, (2) were in a language other than English or French,
or (3) evaluated a tool that (a) consisted of an electronic data-
base with no exportable stand-alone tool (such as an electronic
medical record), (b) had supplementary hardware or software
requirements that were not readily available to the average user,
or (c) required a real-time operator (such as a healthcare
Titles and abstracts were screened for relevance by two inde-
pendent reviewers (CHY, SES; figure 1). Potentially eligible full
articles were then retrieved and reviewed independently by two
reviewers (CHY, SES) to determine whether they met inclusion
criteria. A third reviewer was available in cases of disagreement.
Determination of tool accessibility
Tools were evaluated for accessibility by attempting to access
the web-connected tool on a personal computer with standard
software. If the tools were not readily accessible online, two
attempts were made to contact the authors by email for addi-
tional information or access.
Data extraction and quality assessment
For each study that identified an accessible tool, two reviewers
independently extracted study characteristics using electronic
data extraction forms. We used a modification of the Cochrane
Effective Practice and Organization of Care Group data
abstraction form.22These forms characterized study design,
participants, tool description, study outcomes, and results.
Differences in data extraction were reconciled by consensus.
Tools identified from the grey literature search were catego-
rized by content and educational focus (online appendices). We
randomly selected five websites from each of these categories
and reviewed them to determine if there was evidence of clinical
effectiveness. To assess these sites, we developed an instrument
based on a framework by Straus and Haynes14and tested its face
validity with relevant experts. This instrument contained 18
items and characterized the evidence base for the content and
effectiveness of the tool.
Clinical usefulness and sustainability
Each tool was independently reviewed for clinical usefulness and
sustainability by two members of a team of clinical experts
(CHY F Kim, H Halapy, and C West; see Acknowledgments).
Differences were reconciled by consensus. As there were no vali-
dated instruments to assess clinical usefulness or sustainability,
we developed instruments using a framework from the literature
and input from experts14 23; these instruments were determined
to have face validity by experienced clinicians and experts in
knowledgetranslation. The clinical
contained five items and assessed clinical relevance and ease of
access using a Likert scale with scores ranging from 0 to 5. A score
of 5 denoted ‘clinically useful answers are available most of the
time, and are easily accessible and readable within a few minutes’,
and a score of 0 denoted ‘not useful clinically.’ The sustainability
instrument contained six items and assessed continued relevance
of the topic, potential barriers to sustainability, and engagement
of a group to keep the tool up to date. The instrument was
designed to exclude major barriers to sustainability.
Each tool was independently reviewed for usability by two
members of a team of human factors engineers (S Jovicic, A Xu,
H Takeshita, and F Wan; see Acknowledgments). The instru-
ment incorporated questions from three industry-standard
usability instruments (ISO 9241-110 Usability Heuristics,24
Gerhardt-Powals Research-based Heuristic,25and Site Assess-
ment Tool26) and contained 27 items characterizing suitability
to user’s skill, ease of navigation, reduction in cognitive load,
and appearance. For each desirable usability characteristic, raters
scored ‘yes’, ‘no’, or ‘not applicable.’
Data synthesis and analysis
Inter-rater reliability for data abstraction for clinical effective-
ness, clinical usefulness, sustainability, and usability were
Owing to heterogeneity in study design, population, inter-
ventions, and outcomes, meta-analyses by intervention type or
outcome were not possible. However, we described study quality
and performed a descriptive analysis of studies with evidence
of impact on outcomes. In addition, we performed a meta-
regression of all eligible studies (irrespective of intervention
J Am Med Inform Assoc 2012;19:514e522. doi:10.1136/amiajnl-2011-000307 515
type or outcome) to assess whether clinical usefulness ratings
and usability ratings were moderators of the effectiveness of
these interventions. The meta-regression was performed in R27
version 2.12.0 using the contributed package metafor28version
1.4-0. The Mad29version 0.8 package was used to convert
the treatment effects into a standardized treatment effect.
Hedges G30Studies were excluded if there were insufficient data
to determine a treatment effect or its variance. Studies without
a ‘true’ control group were also excluded, as there is no way to
incorporate clinical usefulness or usability for both groups.
Results of the literature search, study, and tool selection are
detailed in figure 1. While 393 studies and 219 unique tools were
identified, 127 tools were not accessible, and thus we were not
able to evaluate them. We evaluated the remaining 92 tools and
corresponding 57 studies.
Inter-rater reliability was moderate to good: k for data abstrac-
tion items for clinical effectiveness ranged from 0.66 to 0.72.
Weighted k values31for assessment of clinical usefulness and
sustainability were 0.50 (95% CI 0.36 to 0.64) and 0.77 (95% CI
0.63 to 0.87), respectively. Cohen’s k32ranged from 0.50 to 0.65
for each component of the scale (ISO, 0.65; Gerhardt-Powals,
0.50; Site Assessment Tool, 0.55).
Description of studies
Study quality and type
Of 57 studies, 40 studies used experimental designs (25
randomized controlled trials (RCTs),33e56one controlled clinical
trial,5714 uncontrolled beforeeafter studies39 47 58e68), and 17
studies used observational designs (one caseecontrol trial,69
seven cross-sectional studies,70e79nine cohort studies76e78 80e85).
One article consisted of four studies including two RCTs and
two uncontrolled beforeeafter studies.47One article consisted of
one RCTand one uncontrolled beforeeafter study.39
Risk of bias
The methodological quality of all studies is described in online
appendix tables 1 and 2. Characteristics of the RCTs are
summarized in appendix table 1. Methodological quality was
variable; intention-to-treat analysis and description of loss to
Published literature search algorithm. PRISMA flow diagram outlining results of published literature search and tool identification process.
516J Am Med Inform Assoc 2012;19:514e522. doi:10.1136/amiajnl-2011-000307
follow-up occurred in approximately half of the studies, and
calculation of statistical power, randomization, concealment of
allocation, and follow-up of more than 6 months were described
in the minority of studies.
Description of tools
Formats included static websites, decision aids,34 49 52 56 63 64
interactive websites,34e36 38e41 44e46 48 51 54e62 65e76 78 79 81 82 85
CD-ROMgames or DVD,33
programs.34 37 47 52 53 77
Five tools targeted patients with type 1 or 2 diabetes.33 42 55 74 75 79
The remaining tools did not specifically target patients with dia-
betes, but did address an aspect of comprehensive diabetes care in
overweight adults,40 53e55 57smoking adults,35 47 49 51 56 59 60 70 71
76 77 80depressed adults,41 58 69children,43 62smoking adoles-
cents,48 50and adolescents at risk of type 2 diabetes.68With
respect to informal care givers and healthcare professionals,
10 studies targeted healthcare providers, with six targeting
physicians,54 65 72 73 82e84three studies targeting nurses,38 39and
one study targeting public health professionals61; there were no
studies that targeted care givers.
Clinical usefulness, sustainability, and usability
Clinical usefulness, sustainability, and usability ratings are
summarized in table 1. The most common usability error (found
in 50% of tools) was not utilizing images to facilitate learning,
a feature that has been demonstrated to aid data interpretation
and improve recognition and recall.25
Clinical effectiveness of tools
Patient outcomes, including knowledge, skill development,
behavior change, and psychological and clinical outcomes, were
examined in 17 studies.37 40e53 55Clinician knowledge and skill
were evaluated in three studies38 39 54(online appendix table 3).
Patient diabetes education tools
One study examined the effect of a multimedia general diabetes
education computer application for low-literacy patients to use
in clinic waiting rooms.42At 1 year, the intervention group had
a greater awareness of diabetes complications and a greater
reduction in HbA1c than the control group (online appendix
Patient self-management and coping tools
Two studies examined the effect of a self-management website
or DVD in patients with poorly controlled diabetes.33 55In the
first study,55the intervention group had a greater reduction in
HbA1c and systolic blood pressure, an increase in high-density
lipoprotein cholesterol, and reduction in triglycerides compared
with the control group at 12 months (online appendix table 3).
In addition, greater website use correlated with greater clinical
improvements: persistent website users had greater reduction in
HbA1c from baseline compared with intermittent users (?1.9%
vs ?1.2%, p¼0.051). Similarly, larger numbers of website data
uploads were associated with a larger decline in HbA1c (highest
Summary of clinical usefulness, sustainability, and usability ratings
Summary of clinical usefulness ratings
Number of tools for which:
Had clinically useful
Answers were easily accessible
within a few minutes References
Answers were not
easily accessible References
Most of the time 2636 37 46 49 51 52 56 57 60 62
63 65 67 68 72 73 78 82e85
49 51 54 55 57 66
2934 35 39e45 47 48 57 58 64
69 77 79 80
38 57 59 70 71 76 81Some of the time 18
Number of tools
53 57 61
57 74 75
Not at all
Summary of sustainability ratings
Sustainability instrument item
Number of tools with response of:
Barriers identifiedYes References No References
Will this topic continue to be relevant?
Are there any potential barriers for
patients, care givers, the public, or
healthcare providers to using this tool?
Can this tool be easily integrated into
existing practice and systems?
Can groups be easily engaged to facilitate
sustainability of this tool?
Is there a leader responsible for making
modifications to this tool as new
knowledge is brought forward?
34e78 80e83 85
48 53e55 57 59 66 68
2434e47 49e52 56e58
60e65 73 77e85
Length of time required, login
requirement, presence of advertising,
and site credibility
Use of country-specific language and
measurement units and cost
2334e39 42 43 46 47 49e57
60 62e69 72e75 77e80 82e85
34 35 37e39 42 43 46 55e57
60e68 70 72e75 77e80 82e85
34 35 37e40 42e55 57 60e68
70 72e75 77e85
9 40 41 44 45 48 53 57e59
61 68 69 71 76 81
36 40 41 44 45 57e59
69 71 76 81
36 41 58 59 69 71 76
Summary of usability ratings
Number of usability errors Number of toolsReferences
533 42 43 57
34 36 38 39 41 44 45 47 49 51 54e59
61 64e66 68 69 71e77 80e85
37 46 49e52 57 62 63 67 68 70 79
35 40 48 53 55 57 60 68 78
J Am Med Inform Assoc 2012;19:514e522. doi:10.1136/amiajnl-2011-000307 517
tertile ?2.1% vs lowest tertile ?1.0%, p<0.02). The second
study, which compared in-person class-based programs with
DVD-based self-management programs, showed no differences
in self-management behavior or psychological or clinical
One study examined the effect of an interactive website
providing tailored advice on lifestyle modification and risk factor
screening, based on a questionnaire on family history and life-
style habits.36Compared with a control group who received
standardized messages, the intervention group improved their
diet and physical activity, although there were no changes in
smoking rates and assessment of blood glucose or blood pressure,
and a reduction in cholesterol screening.
Patient nutrition and physical activity tools
Seven studies examined the effect of nutrition and physical
activity websites37 40 44e46 52 53on waist circumference,40
weight,53body mass index (BMI),44percentage body fat,44blood
pressure,44quality of life,37and coronary RR.40
Measures of obesity (waist circumference, weight, BMI, and
percentage body fat) were significantly improved,40 44 53as was
and self-assessed health status37
appendix table 3). In a study of a web-based physical activity
program,40waist circumference decreased significantly in the
intervention group compared with the controls. Similarly, an
interactive personalized health promotion website was found to
result in a greater reduction in percentage body fat compared
with the control, although there was no significant change in
BMI or blood pressure.44This study also found a significant
reduction in coronary RR in the intervention group, compared
with the control group. A third study compared the effect of
a weight loss website in combination with human-email coun-
seling, computer-automated email counseling, or no coun-
seling53and found that, at 3 months, there was a significantly
greater weight loss in the human-email group and computer-
automated email group than in the no counseling group.
However, at 6 months, only the human-email group retained
significantly greater weight loss compared with the no coun-
seling group. Finally, self-assessed health status was significantly
greater in the group using an email physical activity and diet
program than the control group.37
One study examined the effect of an educational CD-ROM
game about nutrition and physical activity for children.43
Whereas there was a greater reduction in BMI in girls (p¼0.04),
a greater increase in BMI was noted in boys (p¼0.04) 5 weeks
after the intervention.
Patient smoking cessation tools
Eight studies examined the effect of five online smoking
abstinence,357-day smoking abstinence,34 47 4930-day point
prevalence51 56). Study quality was variable. There was incon-
sistent reporting of loss to follow-up, similarity of groups at
baseline, or adequacy of randomization (online appendix
table 3). It was unclear whether analysis was conducted by
intention-to-treatin the studies
One study of an interactive, multimedia smoking prevention
and cessation curriculum demonstrated lower rate of smoking
initiation compared with use of a self-help booklet.50With
respect to smoking cessation, one study showed no change in
cigarette use at 6 months,48three studies demonstrated no
difference in quit rate (as measured by 1-day reported
abstinence,357-day reported abstinence,34 47 4830-day point
prevalence51 56), and one study showed an improvement in quit
rate.47However, in an exploratory analysis by website utilization,
Rabius found that higher smoking-quit rates were associated
with the two more highly utilized websites compared with the
three less frequently utilized sites (12.5% vs 10.6%, p¼0.03).51
Similarly, participants who visited a site more than five times
were twice as likely to quit than participants who visited a site
less than five times (20.0% vs 9.8%, p<0.001). In addition, higher
quit rates were found with more interactive, tailored sites
compared with the static control site (13% vs 10%, p¼0.04).51
Clinician education tool
One study examined the effect of an online continuing educa-
tion seminar on physician knowledge of diabetes management.54
Physicians’ recommendation of appropriate quality-of-care
measures was assessed immediately after the intervention using
a non-validated clinical vignette score and did not change, with
the exception of one process-of-care measure (‘ordering an eye
Clinician behavior change counseling tool
Two studies examined the effect of an interactive web-based
motivational interviewing educational program on teaching
effectiveness immediately after the intervention.38 39In both
studies, teaching effectiveness, as measured by qualitative
analysis and coding of written responses to counseling vignettes
and a multiple choice questionnaire, was higher in the inter-
Role of interactivity
More interactive tools resulted in greater clinical improvement;
for example, Tate et al found that interaction in the form of
human- or computer-email counseling resulted in greater weight
loss than no counseling.53Similarly, Goran and Reynolds found
that an interactive multimedia CD-ROM game resulted in
greater reduction in BMI than a static educational CD-ROM.43
This observation is seen also in patient smoking cessation
websites; Rabius et al found that interactive tailored smoking
cessation sites resulted in greater quit rates than a static site,51
and Munoz et al found that individually timed educational
messages resulted in greater quit rates than the static smoking
guide alone.47A similar finding was seen in tools for healthcare
providers; Carpenter et al found that an interactive tutorial was
more effective in teaching motivational interviewing techniques
than reading material.38 39The role of tool interactivity on
continued website use is highlighted in the study of the inter-
active personalized health promotion website: Hurling et al
found that the interactive site resulted in a significantly greater
percentage of participants logging in throughout the study
period, with less attrition than the static, control site.45
Tool selection and evaluation are described in figure 2. Of the 360
websites reviewed, two86 87had been evaluated for clinical
effectiveness. Both evaluations had been identified in the pub-
lished literature search47 48and the tools described previously.
Clinical usefulness and usability as potential moderators of tool
Figure 3A,B shows the results of the meta-regression, with
clinical usefulness ratings and usability ratings as potential
moderators of tool effectiveness. Twelve studies were included,
comprising a total sample size of 2731 participants. There was
518J Am Med Inform Assoc 2012;19:514e522. doi:10.1136/amiajnl-2011-000307
significant heterogeneity, with s of 0.87 and 0.85, respectively.
While the standard meta-analysis demonstrated a significant
positive effect on outcomes (standardized treatment effect,
Hedges G 0.64, 95% CI 0.15 to 1.13, p¼0.01), neither clinical
usefulness nor usability had a moderating effect on tool effec-
tiveness (regression coefficient 0.26 (95% CI ?1.4 to 1.9, p¼0.76)
and ?1.5 (95% CI ?6.4 to 3.4, p¼0.55), respectively).
Although a large number of studies and tools were identified,
many tools were not accessible, and thus we were not able to
evaluate them. These tools would also not be accessible to
patients or healthcare providers; thus their exclusion does not
affect the applicability or relevance of our findings. The 57
studies and tools identified were very different in terms of
participants, settings, and outcomes, which meant we could not
perform a meta-analysis by intervention or outcome. Although
there were a number of studies with positive results, these
results must be viewed with caution because of concerns about
the reported study designs. Half of the studies were preepost
designs or included a comparative group that was non-
randomized or not adequately randomized. Many studies would
have been strengthened through use of validated outcome
measures and longer-term follow-up of 1 year or more. With
respect to the tools, although the evidence base of the tool’s
content was high, only 25% had easily accessible, clinically
useful answers most of the time. Six percent of tools were free of
usability errors, but 60% had three or more errors in usability.
Common usability errors included limited use of images, icons,
and other visual elements to facilitate learning, and lack of
intuitiveness in navigation and expected next steps. These and
other usability errors can negatively affect users’ experience with
a tool and may lead them to stop using the tool. In one study,
ease of usability was one of the main determinants of an indi-
vidual’s satisfaction and willingness to engage with a website.88
While a correlation between usability and tool effectiveness was
not demonstrated in this study, our meta-regression was limited
by the number of studies that adequately reported data, as well
as theheterogeneityin interventions,
outcomes. This high prevalence of usability errors is mirrored in
other reviews of usability of healthcare websites89e91and
highlights the need to ensure that websites provide useful and
usable formats and undergo usability testing before they are
Our review of the literature has identified areas for further
exploration. First, greater improvements in patient outcomes
were seen with greater use of the tool.42 49 51 55For example,
persistent website users had greater improvement in HbA1c
than intermittent users, and a larger number of website data
uploads was associated with a larger decline in HbA1c.55
-No evidence identified
that tool was evaluated
-Not further evaluated
1200 websites identified
through Google search
Categorized by tool content
and educational focus
360 tools reviewed for
-Evidence identified that tool
5 tools excluded:
5 tools randomly selected
from each category
diagram outlining results of grey literature search and tool identification
Grey literature search algorithm. Modified PRISMA flow
the observed treatment effects and CI. Grey diamonds show the predicted treatment effects based on the model. (B) Modified forest plot
demonstrating lack of moderating effect of usability ratings on tool effectiveness. Squares with lines are the observed treatment effects and CI. Grey
diamonds show the predicted treatment effects based on the model.
(A) Modified forest plot demonstrating lack of moderating effect of clinical usefulness ratings on tool effectiveness. Squares with lines are
J Am Med Inform Assoc 2012;19:514e522. doi:10.1136/amiajnl-2011-000307519
Caution should be used when interpreting this association, given
possible confounders that can result in reduced use such as
depression or lack of progress with respect to goals. A recent
systematic review found mixed results in the association
between adherence and outcomes; analysis was limited because
of heterogeneity of adherence and outcome measures, although
logins appeared to be associated with outcomes of physical
health interventions, while module completion appeared to be
associated with outcomes of psychological health interven-
tions.92In conjunction with the literature on website usage
attrition,93e95these findings have implications for website
development and website evaluation. Second, this review
suggests a mechanism by which to minimize attrition and thus
maximize clinical improvement, through the use of interactivity
and feedback.44 51 53For example, Hurling et al found that an
interactive health promotion site resulted in a significantly
greater percentage of participants logging in throughout the
study period, with less attrition, than the control static site.44
Although greater interactivity may result in better outcomes, it
may require higher levels of health literacy, navigation skills, and
computer experience.96 97However, with careful user testing,
highly interactive applications can be designed to be user
friendly98and can have positive effects on user satisfaction,
effectiveness, efficiency, and overall attitude toward the tool.99
Other intervention characteristics that enhance use include peer
or counselor support, email or phone contact, and updates
regarding the intervention website.100
These findings have implications for website developers,
researchers, patients, and clinicians. Web-based tool developers
must incorporate strategiesdsuch as optimization of website
usability and interactivitydto maximize frequency and persis-
tence of website use, and researchers must evaluate these
strategies and their impact on website usage and clinical
outcomes, as well as characteristics of users who are predisposed
to persistent website use. Given the degree of variability in
website quality, patients and clinicians should critically appraise
these resources for effectiveness, relevance, and usability before
selecting them for use. Given the burden of time and expertise
required to make these assessments, development of a trans-
parent recognized peer-review system to assess clinical effec-
tiveness, usefulness, sustainability, and usability of web-based
tools, as well as a requirement for standard reporting of these
characteristics by website developers, would enable both
patients and clinicians to make informed decisions in a timely
manner. Although website certification systems do exist, these
primarily address the evidence base of the website content rather
than website use, and do not address clinical usefulness or
usability. For example, Health On the Net Foundation (HON,
http://www.hon.ch/) is a non-profit organization that estab-
lished HONcode certification, an ethical standard aimed at
offering quality online health information. A review of
HONcode-accredited sites found that 87% were too difficult to
read for the average adult population.101In addition, this system
has not been universally adopted, with only 28% of diabetes
patient education sites being HonCODE-accredited.102
This review is limited by the number of accessible tools,
exclusion of tools for mobile devices, the quality of the studies
identified, use of non-validated scales, and publication bias. In
addition, the broad interventions included, as well as the
number of study outcomes (clinical effectiveness, clinical
usefulness, sustainability, and usability), limited the ability to
synthesize data with a standard meta-analytical approach.
Mobile devices represent a highly accessible portal to health
information resources and thus have the potential to transform
healthcare delivery; however, assessing tools for mobile devices
was beyond the scope of this review. Although the rating scales
used were not formally validated, the items were derived from
the literature14 23e26and were assessed for face validity by
content experts. We chose to be inclusive when selecting inter-
ventions, given the multi-system involvement of the diabetic
disease process and the importance of comprehensive manage-
ment (including vascular risk modification) in the care of
the individual with diabetes. We also chose to assess the non-
traditional outcomes of clinical usefulness, sustainability, and
usability, as these are important predictors of knowledge use and
transfer.14 88We strove to reduce publication bias by including
a comprehensive search of the grey literature.
The strengths of this review include: an extensive literature
search that included the grey literature; the comprehensive
review of each tool for clinical effectiveness, usefulness,
sustainability, and usability; and the generalizability of findings
regarding website use in health promotion and chronic disease
management. To our knowledge, although other reviews of
health informatics tools have addressed clinical outcomes12 103
and usability16individually, no other systematic review of any
informatics intervention has considered all of these issues.
Web-based tools have the potential to improve health
outcomes and complement healthcare delivery, but their full
potential is hindered by limited knowledge about their effec-
tiveness, high prevalence of usability errors, and high attrition
rates. A development and research agenda should include:
developing strategies to reduce website attrition in order to
maximize clinical outcomes; standardizing website quality
indicators; and transparent reporting of these indicators in order
to allow patients and clinicians to make informed decisions
about website choice.
Acknowledgments We thank: Laure Perrier for carrying out the published and grey
literature searches; David Newton for information technology support; Julie Hukui,
Daniel Chong, Patrick Ong, and Kristin Tokarsky for reference management;
Mahmood Beheshti, Nathan McKibbon, Alex Rogovic, and Violetta Sochka for
website review; our clinician experts (Florence Kim, Henry Halapy, and Christine
West) for rating clinical usefulness and sustainability; and our human factors
engineers (Sasha Jovicic, Annie Xu, Harumi Takeshita, and Flora Wan) for rating
Funding This research was supported by the Ontario Ministry of Health and Long
Term Care, who had no role in study design, in the collection, analysis and
interpretation of data, in the writing of the report, or in the decision to submit the
article for publication. SES is supported by a Tier 1 Canada Research Chair.
Competing interests None.
Contributors CHY conceived the study, participated in its design, and drafted the
manuscript. RB participated in the coordination, performed data abstraction, and
drafted portions of the manuscript. EL and MSO performed data abstraction and
drafted portions of the manuscript. SES and AL participated in its design and revised
the manuscript critically for important intellectual content. All authors had access to
the data and read and approved the final manuscript.
Provenance and peer review Not commissioned; externally peer reviewed.
Unwin N, Whiting D, Gan D, et al. IDF Diabetes Atlas. 4th edn. International
Diabetes Federation, 2009.
Saadine JB, Cadwell MS, Gregg EW, et al. Improvements in diabetes processes of
care and intermediate outcomes: United States, 1988e2002. Ann Intern Med
Ferlie EB, Shortell SM. Improving the quality of health care in the United Kingdom
and the United States: a framework for change. Milbank Q 2001;79:281e315.
Vermeire E, Wens J, Van Royen P, et al. Interventions for improving adherence to
treatment recommendations in people with type 2 diabetes mellitus. Cochrane
Database Sys Rev 2005;(5):CD003638.
Renders CM, Valk GD, Griffin SJ, et al. Interventions to improve the management
of diabetes mellitus in primary care, outpatient and community settings. Cochrane
Database Sys Rev 2001;(1):CD001481.
520J Am Med Inform Assoc 2012;19:514e522. doi:10.1136/amiajnl-2011-000307
6. Statistics Canada: Canadian Internet Use Survey 2008. http://www.statcan.gc.ca/
adm¼8&dis¼2 (accessed 6 May 2009).
Jeste DV, Dunn LB, Folsom DP, et al. Multimedia educational aids for improving
consumer knowledge about illness management and treatment decisions: a review
of randomized controlled trials. J Psychiatr Res 2008;42:1e21.
Murray E, Burns J, See TS, et al. Interactive Health Communication Applications
for people with chronic disease. Cochrane Database Syst Rev 2004;(4):CD004274.
Seidman JJ, Steinwachs D, Rubin H. Design and testing of a tool for evaluating
the quality of diabetes consumer-information Web sites. J Med Internet Res
Bull SS, Gaglio B, McKay HG, et al. Harnessing the potential of the internet to
promote chronic illness self-management diabetes as an example of how well we
are doing. Chronic Illn 2005;1:143e55.
Banzi R, Liberati A, Moschetti I, et al. A review of online evidence-based practice
point-of-care information summary providers. J Med Internet Res 2010;12:e26.
Gibbons MC, Wilson RF, Samal L, et al. Impact of consumer health informatics
applications. Evid Rep Technol Assess (Full Rep) 2009:1e546.
Mackert M, Kahlor L, Tyler D, et al. Designing e-health interventions for low-
health-literate culturally diverse parents: addressing the obesity epidemic. Telemed
J E Health 2009;15:672e7.
Straus S, Haynes R. Managing evidence-based knowledge: the need for reliable,
relevant and readable resources. CMAJ 2009;180:942e5.
ISO/IEC. Ergonomic Requirements for Office Work with Visual Display Terminals
(VDT)s - Part 11 Guidance on Usability. ISO/IEC 9241-11 1998:(E).
Bock B, Graham AL, Sciamanna C, et al. Smoking cessation treatment on the
Internet: content, quality, and usability. Nicotine Tob Res 2004;6:207e19.
Rogers E. Diffusion of Innovations. 5th edn. New York: Free Press, 2005:429.
Mendel P, Meredith LS, Schoenbaum M, et al. Interventions in organizational and
community context: A framework for building evidence on dissemination and
implementation in health services research. Adm Policy Ment Health
Perry KJ, Hickson M, Thomas J. Factors enabling success in weight management
programmes: systematic review and phenomenological approach. J Hum Nutr Diet
Gaede P, Lund-Andersen H, Parving H, et al. Effect of a multifactorial intervention
on mortality in type 2 diabetes. N Engl J Med 2008;358:580e91.
Canadian Agency for Drugs and Technologies in Health. Grey Matters: A
Practical Tool for Evidence-Based Searching. http://www.cadth.ca/indexphp/en/
cadth/products/grey-matters (accessed 6 May 2009).
Cochrane Effective Practice and Organisation of Care Group. EPOC
Resources for Review Authors. http://epoc.cochrane.org/epoc-resources-
Davies B, Edwards N. The action cycle: sustaining knowledge use. In: Sharon S,
Jacqueline T, Ian DG, eds. Knowledge Translation in Health Care: Moving from
Evidence to Practice. West Sussex, UK: Blackwell Publishing Ltd, 2009.
Ergonomics of Human-System Interaction Part 110: Dialogue Principles. ISO
Gerhardt-Powals J. Cognitive engineering principles for enhancing human
computer performance. Int J Hum Comput Interact 1996;8:189e21.
U.S. Department of Health & Human Services: Site Assessment Tool.
Adapted from Research-Based Web Design & Usability Guidelines. 2009. http://
www.usability.gov/guidelines/ (accessed 13 Aug 2009).
R Development Core Team. A Language and Environment for Statistical
Computing. Vienna, Austria: R Foundation Statistical Computing, 2010, ISBN
Wolfgang V. Conducting meta-analyses in R with metafor package. J Stat Softw
Del Re AC, Hoyt WT. Mad: Meta-Analysis with Mean Differences R package
version 08. 2010.
Cooper H, Hedges LV, Valentine JC. The Handbook of Research Synthesis and
Meta Analysis. New York, US: Russell Sage Foundation, 2009.
Fleiss JL, Cohen J, Everitt BS. Large-sample standard errors of kappa and
weighted kappa. Psychol Bull 1969;72:323e7.
Cohen J. A coefficient for agreement for nominal scales. Educ Psychol Meas
Glasgow RE, Edwards LL, Whitesides H, et al. Reach and effectiveness of
DVD and in-person diabetes self-management education. Chronic Illn
Mun ˜oz RF, Barrera AZ, Delucchi K, et al. International Spanish/English Internet
smoking cessation trial yields 20% abstinence rates at 1 year. Nicotine Tob Res
Etter J. Comparing computer-tailored, internet-based smoking cessation counseling
reports with generic, untailored reports: a randomized trial. J Health Commun
Ruffin MT 4th, Nease DE Jr, Sen A, et al. Effect of preventive messages tailored to
family history on health behaviors: The Family Healthware Impact Trial. Ann Fam
Block G, Sternfeld B, Block CH, et al. Development of Alive! (A lifestyle intervention
via email), and its effects on health-related quality of life, presenteeism, and
other behavioral outcomes: Randomized controlled trial. J Med Internet Res
Carpenter KM, Cohn LG, Glynn LH, et al. Brief interventions for smoking cessation:
Using the internet to train healthcare providers. Int Electron J Health Educ
Carpenter KM, Watson JM, Raffety B, et al. Teaching brief interventions for
smoking cessation via an interactive computer-based tutorial. J Health Psychol
Carr LJ, Bartee RT, Dorozynski C, et al. Internet-delivered behavior change program
increases physical activity and improves cardiometabolic disease risk factors in
sedentary adults: Results of a randomized controlled trial. Prev Med
Christensen H, Griffiths KM, Jorm AF. Delivering interventions for depression by
using the internet: Randomized controlled trial. BMJ 2004;328:265e9.
Gerber BS, Brodsky IG, Lawless KA, et al. Implementation and evaluation of a low-
literacy diabetes education computer multimedia application. Diabetes Care
Goran MI, Reynolds K. Interactive multimedia for promoting physical activity
(IMPACT) in children. Obes Res 2005;13:762e71.
Hurling R, Catt M, Boni MD, et al. Using internet and mobile phone technology to
deliver an automated physical activity program: Randomized controlled trial. J Med
Internet Res 2007;9:e7.
Hurling R, Fairley BW, Dias MB. Internet-based exercise intervention systems: Are
more interactive designs better? Psychol Health 2006;21:757e72.
Martinson BC, Crain AL, Sherwood NE, et al. Maintaining physical activity among
older adults: Six-month outcomes of the keep active Minnesota randomized
controlled trial. Prev Med 2008;46:111e19.
Munoz RF, Lenert LL, Delucchi K, et al. Toward evidence-based internet
interventions: A Spanish/English web site for international smoking cessation trials.
Nicotine Tob Res 2006;8:77e87.
Norman CD, Maley O, Li X, et al. Using the internet to assist smoking prevention
and cessation in schools: a randomized, controlled trial. Health Psychol
Pike KJ, Rabius V, McAlister A, et al. American cancer society’s QuitLink:
randomized trial of internet assistance. Nicotine Tob Res 2007;9:415e20.
Prokhorov AV, Kelder SH, Shegog R, et al. Impact of a smoking prevention
interactive experience (ASPIRE), an interactive, multimedia smoking prevention and
cessation curriculum for culturally diverse high school students. Nicotine Tob Res
Rabius V, Pike KJ, Wiatrek D, et al. Comparing internet assistance for smoking
cessation: 13-month follow-up of a six-arm randomized controlled trial. J Med
Internet Res 2008;10:e45.
Sternfeld B, Block C, Quesenberry CP, et al. Improving diet and physical activity
with alive: a worksite randomized trial. Am J Prev Med 2009;36:475e83.
Tate DF, Jackvony EH, Wing RR. A randomized trial comparing human e-mail
counseling, computer-automated tailored counseling, and no counseling in an
internet weight loss program. Arch Intern Med 2006;166:1620e5.
Weston CM, Sciamanna CN, Nash DB. Evaluating online continuing medical
education seminars: Evidence for improving clinical practices. Am J Med Qual
McMahon GT, Gomes HE, Hohne SH, et al. Web-based care management in
patients with poorly controlled diabetes. Diabetes Care 2005;28:1624e9.
Graham AL, Cobb NK, Papandonatos GD, et al. A randomized trial of Internet and
telephone treatment for smoking cessation. Arch Intern Med 2011;171:46e53.
Woolf SH, Krist AH, Johnson RE, et al. A practice-sponsored web site to help
patients pursue healthy behaviors: an ACORN study. Ann Fam Med
Christensen H, Griffiths KM, Korten A. Web-based cognitive behavior therapy:
analysis of site usage and changes in depression and anxiety scores. J Med Internet
Graham AL, Cobb NK, Raymond L, et al. Effectiveness of an internet-based
worksite smoking cessation intervention at 12 months. J Occup Environ Med
Thieleke J, McMahon J, Meyer G, et al. An evaluation of the freedom from
smoking online cessation program among Wisconsin residents. WMJ
Sears KE, Cohen JE, Drope J. Comprehensive evaluation of an online tobacco
control continuing education course in Canada. J Contin Educ Health Prof
Bell JA, Patel B, Malasanos T. Knowledge improvement with web-based diabetes
education program: Brainfood. Diabetes Technol Ther 2006;8:444e8.
VanWormer JJ, Pronk NP, Boucher JL. Experience analysis of a practice-based,
online pedometer program. Diabetes Spectrum 2006;19:197e200.
Lalonde L, O’Conner AM, Drake E, et al. Development and preliminary testing of
a patient decision aid to assist in pharmaceutical care in the prevention of
cardiovascular disease. Pharmacotherapy 2004;24:909e22.
Wells S, Furness S, Rafter N, et al. Integrated electronic decision support increases
cardiovascular disease risk assessment four fold in routine primary care practice.
Eur J Cardiovasc Prev Rehabil 2007;15:173e8.
Smith KE, Levine BA, Clement SC, et al. Impact of MyCareTeam for poorly
controlled diabetes mellitus. Diabetes Technol Ther 2004;6:828e35.
Speck RM, Hill RK, Pronk NP, et al. Assessment and outcomes of HealthPartners
10,000 Steps program in an academic work site. Health Promot Pract
J Am Med Inform Assoc 2012;19:514e522. doi:10.1136/amiajnl-2011-000307521
Long JD, Armstrong ML, Amos E, et al. Pilot using world wide web to prevent
diabetes in adolescents. Clin Nurs Res 2006;15:67e79.
Christensen H, Griffiths KM, Korten A, et al. A comparison of changes in anxiety
and depression symptoms of spontaneous users and trial participants of a cognitive
behavior therapy website. J Med Internet Res 2004;6:e46.
Balmford J, Borland R, Benda P. Patterns of use of an automated interactive
personalized coaching program for smoking cessation. J Med Internet Res 2008;10:
Cobb NK, Graham AL, Bock BC, et al. Initial evaluation of a real-world internet
smoking cessation system. Nicotine Tob Res 2005;7:207e16.
Riddell T, Lindsay G, Kenealy T, et al. The accuracy of ethnicity data in primary care
and its impact on cardiovascular risk assessment and management: Predict CVD-8.
N Z Med J 2008;121:40e8.
Riddell T, Kenealy T, Wells S, et al. Audit of health data captured routinely in
primary healthcare for the clinical decision support system predict (PREDICT CVD-4).
Zrebiec J. Internet communities: do they improve coping with diabetes? Diabetes
Educ 2005;31:825e8, 830e2, 834, 836.
Zrebiec JF, Jacobson A. What attracts patients with diabetes to an internet support
group? A 21 month longitudinal website study. Diabet Med 2001;18:154e8.
An LC, Schillo BA, Saul JE, et al. Utilization of smoking cessation information,
integrative and online community resources as predictors of abstinence: Cohort
study. J Med Internet Res 2008;10:e55.
Lenert L, Munoz R, Perez JE, et al. Automated e-mail messaging as a tool for
improving quit rates in an internet smoking cessation intervention. J Am Med Inform
Petersen R, Sill S, Lu C, et al. Effectiveness of employee internet-based weight
management program. J Occup Environ Med 2008;50:163e71.
Herrejon K, Hartke JL, Scherer J, et al. The creation and impact evaluation of “Your
guide to diet and diabetes,” an interactive web-based diabetes tutorial. Diabetes
Technol Ther 2009;11:171e9.
Lenert L, Munoz RF, Stoddard J, et al. Design and pilot evaluation of an internet
smoking cessation program. J Am Med Inform Assoc 2003;10:16e20.
Ware LJ, Hurling R, Bataveljic O, et al. Rates and determinants of uptake and use of
an internet physical activity and weight management program in office and
manufacturing work sites in England: Cohort study. J Med Internet Res 2008;10:e56.
Marshall RJ, Zhang Z, Broad JB, et al. Agreement between ethnicity recorded in
two New Zealand health databases: Effects of discordance on cardiovascular
outcome measures (PREDICT CVD3). Aust N Z J Public Health 2007;31:211e16.
Selak V, Wells S, Whittaker R, et al. Smoking status recording in GP electronic
records: The unrealised potential. Inform Prim Care 2006;14:235e41; discussion
Whittaker R, Bramley D, Wells S, et al. Will a web-based cardiovascular disease
(CVD) risk assessment programme increase the assessment of CVD risk factors for
maori? N Z Med J 2006;119:U2077.
Sarna L, Bialous S, Wewers ME, et al. Nurses trying to quit smoking using the
Internet. Nurs Outlook 2009;57:246e56.
The Stop Smoking Center. Welcome to the Stop Smoking Center 6.1!. http://
wwwstopsmokingcenternet/ (accessed 1 Mar 2010).
The American Cancer Society. Do You Need Help to Quit? 2008. http://
sitearea¼&level¼ (accessed 1 Mar 2010).
Fu L, Salvendy G. The contribution of apparent and inherent usability to a user’s
satisfaction in a searching and browsing task on the Web. Ergonomics
Seidman JJ, Steinwachs D, Rubin H. Design and testing of a tool for evaluating the
quality of diabetes consumer-information web sites. J Med Internet Res
Sutherland LA, Wildemuth B, Campbell MK, et al. Unraveling the web: an
evaluation of the content quality, usability, and readability of nutrition web sites. J
Nutr Educ Behav 2005;37:300e5.
Bock BC, Graham AL, Whiteley JA, et al. A review of web-assisted tobacco
interventions (WATIs). J Med Internet Res 2008;10:e39.
Donkin L, Christensen H, Naismith SL, et al. A systematic review of the impact of
adherence on the effectiveness of e-therapies. J Med Internet Res 2011;13:e52.
Gunther E. The law of attrition. J Med Internet Res 2005;7:e11. http://
Christensen H, Mackinnon A. The law of attrition revisited. J Med Internet Res
Gunther E. The law of attrition revisited: Author’s Reply. J Med Internet Res
Morrel R. Older Adults, Health Information and the World-Wide Web. New Jersey,
US: Lawrence Erlbaum Associates, 2001.
Milne SAD, Carmichael A, Sloan D, et al. . Are guidelines enough? An introduction
to designing web sites accessible to older people. IBM Syst J 2005;43.
Boquete L, Rodrı ´guez-Ascariz JM, Amo-Usanos C, et al. User-Friendly Cognitive
Training for the Elderly: A Technical Report. Telemed J E Health 2011;17:456e60.
Teo HH, Oh LB, Liu C, et al. An empirical study of the effects of interactivity on web
user attitude. Int J Hum Comput Stud 2003;58:281e305.
Brouwer W, Kroeze W, Crutzen dNJ R, et al. Which intervention characteristics
are related to more exposure to internet-delivered healthy lifestyle promotion
interventions? A systematic review. J Med Internet Res 2011;13:e2.
Kusec S, Brborovic O, Schillinger D. Diabetes websites accredited by the Health On
the Net Foundation Code of Conduct: Readable or Not? Stud Health Technol Inform
Thakurdesai PA, Kole PL, Pareek RP. Evaluation of the quality and contents of
diabetes mellitus patient education on Internet. Patient Educ Couns
Austin Boren S, Gunlock TL, Krishna S, et al. Computer-aided diabetes education:
a synthesis of randomized controlled trials. AMIA Annu Symp Proc 2006:51e5.
PAGE fraction trail=8.75
522 J Am Med Inform Assoc 2012;19:514e522. doi:10.1136/amiajnl-2011-000307