Randomized Study ofOnline Vaccine Reminders in Adult Primary Care
James R. Flanagan, M.D., Ph.D. ab, Bradley N. Doebbeling, MD, MSbcd, JeffDawson, ScDC,
SusanBeekmann, R.N., M.P.H.b
'Division ofClinical Informatics, Information Systems Department, University ofIowa
Hospitals and Clinics, bDepartmentofInternal Medicine, University ofIowa, 'Department of
Preventive Medicine, University ofIowa, dVeterans Affairs Medical Center, Iowa City, Iowa.
Online immunization reminders were implemented in
an adult medicine setting in which all immunization
history, vaccine ordering and charting were required
Physicians were randomized to one of two
arms in a cross-over design. Each arm was shown
online recommendations for vaccines indicated by
nationally accepted guidelines either during thefirst
or during the second part of the study period The
mainpurpose ofthe study was to assess the impact of
reminders on correct decisions related toprescribing
vaccines. Online reminders had thefollowing impact
application almost 3 times as often when shown
reminders. 2)Physicians in the reminder group were
27% less likely to order a vaccine in the reminder
guidelines was improved significantly for Tetanus
and for Hepatitis B in several analyses.
effects were found for Pneumoccocal, Measles, or
pneumococcal pneumonia and hepatitis B cause an
estimated 50,000 excess deaths per year in the United
Considerable data exists regarding the
vaccines. However, the majority of adults for whom
immunization and early detection of disease through
screening and improved case-finding, have led to
Previous observations in our ambulatory care clinics
had indicated that tetanus in particular was highly
is recommended are not immunized.
published trials of educational interventions in the
health care professions to determine the effectiveness
of different types of interventions in improving health
Dissemination-only strategies demonstrated little or
no changes in health professional behavior or health
outcome when used alone. However, more complex
and health outcomes4.
interventions available that, if used appropriately,
could lead to important improvements in professional
There have been few well-designed clinical trials on
the effect of feedback and reminders on preventive
activities in ambulatory practice5. The interventions
reporting of the results
However, the literature demonstrates a positive effect
of feedback on compliance
In this respect, the effect of reminders
may be greater than that of feedback.
important need for research on the effect of feedback,
reminders and other instruments of quality assurance
different clinical settings5.
is insufficiently precise.
There is an
The Immunization application is part of the UIHC
online patient record, INFORMM Patient Record
(IPR), a Windows-based application developed at the
UIHC. The present version ofIPR includes structured
documentation of most of the summary components
of the record (Allergies, Immunization History
Vaccine Orders / Vaccine Charting, Medications /
Prescriptions) and most nursing documentation. All
exam rooms, work rooms, and nursing stations have
Windows devices (PCs or network workstations) for
access to IPR as well as to a number of Web-based
Nearly all (>99%) of vaccine history,
orders, and charting of administration
online. The few written orders are back-loaded into
immunization history, vaccine orders, and vaccine
administration, following the work flow of several
disciplines. The components are: a clinical help file,
a rules database, the historical information on a
patient's vaccines, the process of ordering a new
vaccine, charting the administration ofa new vaccine,
and reports on patients who need vaccines. These are
all described in more detail in another report3.
1091-8280/99/$5.00© 1999 AMIA, Inc.
Information for International Travel (Centers for
Disease Control and Prevention), the Red Book
(American Academy of Pediatrics), and Guide for
Physicians), and U.S. Preventive Services Task Force
Guidelines. Using these sources, we compiled a set
assessment of immunity.
The rules engine generated recommendations for
vaccine orders.Pediatric and adult (>16 years of
age) rules were abstracted from these sources.
entire set was described in detail3. The adult rules are
Hepatitis B was flagged "consider" ifthe patient was
a hospital employee or if less than 25 years of age.
Influenza was recommended during October-January
and the patient was a hospital employee or was
greater than 64.5 years ofage
This was flagged "consider" if born after
1956 and fewer than 2 doses documented and more
than 30 days since most recent dose.
This was recommended if age was
greater than 64 years (flagged "consider" if more
than 63.5 years) and more than 10 years since last
received (flagged "consider" if more than 7 years
since last received).
Td was recommended
vaccine in over 9 years and 6 months.
if no history of Tetanus
evaluated based upon the patient's age and history of
recommendations depends on an accurate patient
vaccine history. Historical information was loaded
into the system from UIHC pharmacy billing records.
supplemented the online history information.
recommended based on the history and the rules.
Also shown were any warnings generated from the
rules engine regarding any ofthe vaccines
recommendation, to order the recommended vaccine,
or to order other vaccines.
reviewed by the person administering the vaccines
who then charted administration.
vaccine history to be automatically updated.
Pending orders were
This caused the
Study design:. All providers working in the General
Medicine Services of the UIHC were randomly
assigned to one of two arms after being stratified by
level of experience prior to randomization. A cross-
over design allowed those in one arm to see the
recommendations only during the first half of the
recommendations only during the second half of the
An automated log ofeach session in which a provider
accessed the Immunization application recorded the
status of vaccine recommendations at the beginning
and the end of the session and whether or not the
physician generated an order for the vaccine during
The Immunization application was introduced to all
including 24 nursing staff, 120 staff physicians, and
113 resident physicians.
given 2 hours training in a computer laboratory as
well as on-the-job training during the first month of
use. Staff and resident physicians were given 1 hour
of training in a tutorial set up in the clinic in which
instructed to review the online immunization history
during the patient visit.
The nursing staff were
In addition, assistance was provided
first month of use. Physicians were
One month after the Immunization application was
introduced into clinic, those in the reminder arm were
first shown recommendations.
arms were informed about the availability of an
online educational resource on vaccines and that
some of them were being given computer generated
reminders while others were not. After five months,
providers in the reminder and control arms were
crossed-over to the alternate group for the remaining
four months of the study
Providers in both
. The cross over date was
Statistical analysis: For analysis of sessions, 2 x 2
contingency table analysis was performed with Chi-
appropriate to compute two-tailed P-values.
exact text where
Since ordering behavior was not expected to be
independent at the physician level, in some analyses
the data were collated by physician. Specifically, the
proportion of physicians who ordered a vaccine at
least once was calculated.
sessions in which a vaccine was ordered (the median
usage rate) was calculated for each physician.
Also, the proportion of
Because of clinical schedule rotations throughout the
year, only twenty-eight of the physicians participated
in both study groups, i.e., crossed over, while 61
physicians were in one group only.
For those that
crossed over, paired analyses were performed.
example, to determine whether the proportion of
physicians who ordered the vaccine at least once
varied across groups, an exact McNemar's test was
A Wilcoxon signed rank test was used to
compare the proportion of opportunities when
vaccine was ordered. For the physicians who did not
Fisher's exact test was used instead ofthe McNemar's
test, and the Wilcoxon Rank-sum test was used
approximation combined the test results to obtain an
omnibus test for all 89 physicians.
There were 1,985 Immunization application sessions
involving non-physicians and 980 sessions involving-
Non-physicians used the application at
least once for each of 1,678 visits involving 1,548
patients. These sessions were used for updating the
in the application
automated reminders to the physician would reflect a
current immunization history.
application for 886 visits (53% ofthe total) involving
Physicians used the
One-fourth(30) of the 120 staff physicians used the
application at least once during the study.
half(55, 49%) ofthe 1 13 resident physicians used the
application at least once. Among 980 physician uses,
there were 103 sessions in which there were no rule-
generated vaccine indications.
sessions the application rules found at least one
These recommendations were
shown to physicians in the reminder arm.
In the remaining
Impact of Reminders:
physicians in the reminder arm (Table 1). The high
fraction of use by physicians in the reminder arm
remained consistent before and after the cross-over,
both for all physician sessions and for the subset of
sessions restricted to physicians who participated in
In both cases the data showed that
physicians used the application much more often
when in the reminder arm.
Nearly three-quarters (726,
There was no order for any vaccine in 420 (43%) of
vaccine in 560 (57%) sessions. There were orders for
at least 2 vaccines in 153 (16%) sessions.
sessions involving physicians in the reminder arm
were less likely to involve an order for a vaccine (P
value <0.0005, RR 0.73, C195 0.60 to 0.88).
, there were orders for at least one
Reminders were provided only if the physician chose
to use the Immunization application.
As such, an
impact on second-vaccine ordering by physicians
who've already decided to order one vaccine was
considered. Among sessions with at least one order,
those sessions with physicians in the reminder arm
were not more likely to involve orders oftwo or more
Sessions by Physicians Who Crossed-over
Effect of reminders on the number of
vaccines overall (Table
significant effects of reminders on the distribution of
specific vaccine orders (Table 2). The table shows the
number of sessions (Total) and the number of
The proportion of sessions
specific vaccine ordering increased for tetanus and
However, there were
Table 2 Analysis of Sessions in which One or More
Vaccines were Ordered.
Orders for vaccines were classified "correct" if the
vaccine was indicated by the rules and not ordering a
At least 1
P < 0.0005, RR 0.70
(CI95 0.56 - 0.86)
Sessions byAll Physicians
At Least 1
P < 0.0005, RR 0.73
Sessions byAll Physicians
| At least 2
vaccine as "correct" if the vaccine was not indicated
by the rules.
Among all sessions, analysis of the
demonstrated no significant effect of reminders.
However, subset analysis of sessions in which one or
more vaccine was ordered did detect a significant
effect of reminders on correct vaccine decisions for
The analyses of data from
sessions in which at least one vaccine order and for
those involving at least two vaccine orders indicated
a significant effect of reminders on correct decisions
The analysis of the no-order
sessions showed no effect of reminders on the
proportion of correct decisions.
number of no-order sessions is, no doubt, the reason
why no significant effect was observed on correct
ordering for sessions overall.
Given the large
Table 3. Analysis of correct vaccine decisions for all
Table 4. Analysis ofTetanus Decisions
At least 2
Because the physician was the unit ofrandomization,
the data were analyzed for impact on individual
physician behavior with respect to ordering specific
1) the proportion of physicians who
ordered at least once and 2) the median usage rate by
There were no significant
effects on the latter outcome.
demonstrated the proportion ordering a given vaccine
at least once when in the reminder or in the control
arm. Among all physicians, there was a significant
The data (Table 5)
effect on ordering for Hepatitis B (P < 0.004) and a
borderline non-significant effect (P < 0.089) on
ordering for Tetanus. Among physicians who were
in both arms ofthe study, the effect on both vaccines
achieved significance (P < 0.016).
Table 5 Analysis of the effect of reminders on the
proportion of physicians who ordered at least once:
data for all physicians
Table 6. Analysis of the effect of reminders on the
proportion of physicians who ordered at least once:
data restricted to physicians that crossed over.
nursing notes, and all test results were available
online. On the whole, the computer was viewed as
an essential tool in the patient care process.
prescriptions online and over 90% of prescriptions
were written using the computer.
application was the first computer application used in
this setting that was required in order to accomplish a
specific task: the application was required in order to
get any patient vaccinated.
reminders arm saw vaccine reminders every time
they chose to use the Immunization application to
write any vaccine order. All physicians were trained
and able use an online help file to see the vaccine
recommendations on which rules were based.
therefore, the frequency that the reminders were
The effectiveness of online reminders was
dependent on being viewed by a physician, but the
reminders themselves seem to be an inducement to
use the system.
utilization of the application and,
Disappointingly, we did not find an effect on correct
ordering for any vaccine
Certainly this was not because of high
proportions of correct decisions in the absence of
reminders. Since there were a number of impacts of
The fact that recommendations were
often not followed may reflect disagreement with the
information available to the physician, such as patient
history, but not entered into the application would
led to recommendations that the physician
would not have followed.
in analysis of the data
it seems clear that physicians saw the
in the rules. Alternatively,
Insufficient patient history
faced by this kind of application.
every attempt to ensure an accurate online history,
including review of billing records and of the page in
the paper record that was supposed to contain the
summary of immunization history.
demonstrate vaccine compliance based upon data that
were easily found in the record3. The application
described in that report is expected to alleviate that
problem over time through increased reliance on
Immunization application fore all aspects of vaccine-
related health care.
is a common problem
We had made
In a companion
One impact of reminders was the reduction in the
proportion of sessions that resulted in an online order.
reminders group more often used the sessions purely
to gain information,
generated recommendations. The other interpretation
dissuaded the physician from generating an order.
This interpretation is the justification for our having
analyzed the no-order subset of sessions for evidence
of vaccine-specific effects on "correct" decisions.
is that the physicians
this case the computer-
Another subset analysis, the proportion of sessions in
which a second vaccine was ordered among sessions
in which at least one was ordered, was justified by
the fact that a physician would see the reminders only
if that physician chose to use the Immunization
application with the intent of entering a vaccine
order. While subset analyses can result in finding
spurious associations, both of those presented here
had good justifications.
magnitudes were clinically significant.
The resulting associations
The main outcome planned for the study was to
assess the impact of reminders on correct decisions
related to prescribing vaccines. The results of several
analyses showed that there was a statistically and
clinically significant impact on ordering for Tetanus
found to be by far the lowest among all vaccines
studied3. This may in turn reflect the fact that tetanus
and diphtheria are disease rarely seen by the primary
care physician in this country.
vaccination history and,
significant impact on improving compliance with
in selected cases, had a
'McGinnis JM, Foege WH. Actual causes of death in
the United States. JAMA. 1993;270:2207-12.
2US Department of Health and Human Services. Put
Prevention into Practice:
Government Printing Office; 1994:
3Flanagan JR. Walker KP.
Compliance in a Primary Care Setting Using Online
History, Reminders, and Order Entry. (submitted
Proceedings AMIA Fall Symposium 1999)
4Oxman AD, Thomson MA, Davis DA, Haynes RB.
No magic bullets: a systematic review of 102 trials of
interventions to improve professional practice. Can
Med Assoc J. 1995;153:1423-31.
5Buntinx F, Winkens R, Grol R, Knottnerus JA.
Influencing diagnostic and preventive performance in
care by feedback and reminders. A
review. Fam Pract. 1993;10:219-28.
Clinicians Handbook of
Evaluation of Vaccine