Case Report ?
Use of Clinical Decision Support to Increase Influenza
Vaccination: Multi-year Evolution of the System
MARY N. GERARD, MD, WILLIAM E. TRICK, MD, KRISHNA DAS, MD, MARJORIE CHARLES-DAMTE, RN,
GREGORY A. MURPHY, RN, IRENE M. BENSON, APN-CNS
A b s t r a c t Despite recognition that clinical decision support (CDS) can improve patient care, there has been
poor penetration of this technology into healthcare settings. We used CDS to increase inpatient influenza
vaccination during implementation of an electronic medical record, in which pharmacy and nursing transactions
increasingly became electronic. Over three influenza seasons we evaluated standing orders, provider reminders,
and pre-selected physician orders. A pre-intervention cross-sectional survey showed that most patients (95%) met
criteria for vaccination. During our intervention, physicians were increasingly likely to accept pre-selected
vaccination orders, Year 1 (47%), Year 2 (77%), Year 3 (83%); however vaccine administration by nurses was
suboptimal. As electronic medical record functionality improved, patient receipt of vaccine increased dramatically,
Year 1 [0/36; 0%], Year 2 [8/66; 12%], Year 3 [286/805; 36%]. Successful use of clinical decision support to increase
inpatient influenza vaccination only occurred after initiation of CPOE for all medications and integration of an
electronic medication administration record. Also, since most patients met criteria for influenza vaccination,
complicated logic to identify high-risk patients was unnecessary.
? J Am Med Inform Assoc. 2008;15:776 –779. DOI 10.1197/jamia.M2698.
Many improvements in patient-care derived from imple-
mentation of electronic medical records result from use of
clinical decision support (CDS) systems triggered by com-
puterized physician order entry (CPOE).1– 4 Despite the
benefits realized from use of electronic medical records,
CPOE, and CDS, penetration of these systems in U.S. health-
care settings remains low.4–6 Increased adoption of CDS
likely will follow demonstrations of successful implementa-
tions in institutions that have vendor-provided solutions.
Although influenza vaccination effectively prevents disease
and reduces the risk of hospitalization, many high-risk
people are not immunized.7–9 As such, the Joint Commission
on Accreditation of Healthcare Organizations measures in-
fluenza vaccination of eligible pneumonia inpatients as an
indicator of good quality healthcare. For these reasons, and
because the population served by Stroger Hospital (formerly
Cook County Hospital) historically has been undervacci-
nated,10 we designed a CDS-based intervention to increase
influenza vaccination of hospital patients. Since components
of our electronic medical record were introduced incremen-
tally over three influenza seasons; we evaluated use of CDS
during the maturation of our system. We report our chal-
lenges and successes during the implementation of our
Setting and Project Description
Stroger Hospital is a 464-bed public hospital. In 2001, we
began installation of a new information system (Cerner Inc.,
Kansas City, MO.). Our information system gradually
evolved as components were transferred from paper-based
to electronic systems. We used the CDS system to improve
influenza vaccination during the 2003–2004, 2005–2006, and
2006–2007 influenza seasons, i.e., Year 1, Year 2, and Year 3,
respectively. Due to a nation-wide influenza vaccine short-
age, there were no interventions during 2004–2005. We
focused our intervention on internal medicine ward pa-
tients, who were admitted to one of three separate teams;
admissions were consecutively assigned to a team, resulting
in similar patient characteristics between teams. There were
no education sessions for physicians during the three influ-
enza seasons. We obtained approval from the IRB.
Year 1: 2003–2004 Influenza Season
During Year 1, we performed a cross-sectional survey to
determine how often patients met high-priority category
criteria for receipt of influenza vaccination;11 we performed
Computerized provider order entry (CPOE) was available for
laboratory tests, radiographic studies, diets, electrocardiograms,
and admission/discharge orders; there was no medication
CPOE. Nursing documentation was paper-based, including
medication administration. Electronic orders were printed at
the nursing station. No standing orders policy existed for
influenza vaccination (Table 1).
The cross-sectional survey showed that ?95% of internal
medicine patients met a high-priority criterion for vaccina-
tion; therefore, our CDS rule targeted all internal medicine
Affiliation of the authors: Cook County Bureau of Health Services,
Correspondence: William E. Trick, MD, Collaborative Research
Unit, Administration Building, 1900 W. Polk St, Suite 1600, Chicago,
IL, 60612; e-mail: ?email@example.com?.
Gerard et al., Using Clinical Decision Support to Increase Flu Vaccination
inpatients. We tested separate strategies for each internal
medicine team, as follows: 1) A pre-selected order “Admin-
ister Influenza Vaccine 0.5 ml IM” was presented to physi-
cians in real time as a pop-up, triggered by their “Discharge
Patient” order; 2) a pop-up reminder to order vaccine was
presented to physicians upon entry of the “Discharge Pa-
tient” order; 3) no intervention, i.e., usual care.
Year 2: 2005–2006 Influenza Season
During Year 2, we augmented CDS rules with a written
universally-applied standing-orders policy that enabled
nurses to vaccinate inpatients even if there wasn’t a patient-
specific physician’s order. We educated nurses about the
policy and importance of influenza vaccination. Nurses
viewed electronic orders through a patient-centered task list.
Compared to Year 1, physicians entered more orders elec-
tronically, and increasingly documented patient evaluations
in the electronic medical record. However, medication order
entry still was paper-based (Table 1).
We compared the following strategies concurrently (i.e.,
one strategy per team): 1) A pre-selected order “Admin-
ister Influenza Vaccine 0.5 ml IM” was presented to
physicians, triggered by the “Discharge Patient” order; 2)
an electronic reminder to follow the standing orders
policy populated nurses’ task lists upon patient admis-
sion; 3) usual care.
Year 3: 2006–2007 Influenza Season
During Year 3, medication CPOE was active and orders
were routed electronically to the electronic medication ad-
ministration record (Table 1). Since despite physician orders,
vaccine administration was uncommon during Year 2, in
Year 3 we intended to test triggering the order at the time of
patient admission. Unfortunately, we were unable to imple-
ment CDS using the “Admission to Bed” order; therefore,
for Year 3, all results were for the “Discharge Patient”
order—the same intervention as during Year 2.
Influenza Vaccination Assessment
During all three seasons, the pre-selected influenza vaccine
order was enacted unless the physician de-selected the
order. During Years 1 and 2, to determine patient vaccina-
tion receipt, we reviewed paper medication administration
records and nursing notes on a random sample of patients.
During Year 3, since vaccine administration was only doc-
umented electronically, we retrieved these data from the
hospital information system for all patients; patient sam-
pling was unnecessary.
To demonstrate changes in vaccination rates over time—
during the transition from a hybrid paper-electronic system
to an electronic system—we present the results from the
intervention strategy that remained constant (i.e., pre-se-
lected order triggered by the “Discharge Patient” order). We
tested the trend in vaccination receipt by calculating the
chi-square test for trend. All analyses were performed using
Stata version 9.2 (StataCorp, College Station, TX).
From the cross-sectional survey, approximately one-third of
patients were vaccinated before hospitalization; reactions to
eggs or previous influenza vaccination were rare and most
patients met criteria that placed them in a high-priority cate-
gory for vaccination (Table 2).11The most common criterion
met was age ?49 years followed by diabetes, cardiac, pulmo-
nary, or renal disease. For seven patients (7%) the sole
criterion met could only be discerned through bedside
interview; for example, living with a high-risk person or
child aged less than two years.
During Year 1, of 114 patients sampled for chart review,
none were vaccinated. Of 36 patients admitted to the pre-
selected order team, 17 (47%) times the physician accepted
Table 1 y Interventions Used to Increase Vaccination of Hospital Patients and Information System Maturation
over Three Influenza Seasons
Clinical decision support
Standing orders policy# initiated
Standing orders policy continued
Electronic reminder to physicians
to order vaccine
No CDS intervention
Electronic nursing reminder† to follow
standing orders protocol
Pre-selected order triggered by
“Discharge Patient” order
No CDS intervention
Pre-selected order triggered by
“Admission to Bed” order‡
Pre-selected order triggered by
“Discharge Patient” order
Both of the above‡
Team 3Pre-selected order triggered by
“Discharge Patient” order
Laboratory, diet, radiology, EKG,
and discharge orders
electronic; these orders printed
at the nursing station.
Nurses: kardex and patient activity list
Physicians: increased electronic
charting, electronic consult orders.
Nurses: medication orders
Physicians: CPOE available for
CDS ? clinical decision support; EKG ? electrocardiogram; e-MAR ? electronic medication administration record; CPOE ? computerized
physician order entry.
*Mandatory small-group sessions; we educated nurses about the influenza vaccination and the new standing orders policy.
†An automated electronic reminder populated the nurses’ kardex and patient activity list at the time of admission.
‡The clinical decision support system was not fully operable for the “Admission to Bed” order; therefore, these results are not presented.
#The standing orders policy enabled nurses to administer influenza vaccine to patients without an individual physician’s order.
Journal of the American Medical Informatics AssociationVolume 15Number 6 November / December 2008
the order; however, the vaccine never was administered,
Of 204 patients sampled, 66 were admitted to the pre-
selected order medical team. Compared to Year 1, physi-
cians were much more likely to accept the pre-selected order
(77% vs. 47%; p ? 0.002) and patients were more likely to
receive vaccine (8/66 [12%] vs. [0/36] 0%, p ? 0.03), Figure
1. Despite having a standing orders policy, few patients
admitted to the usual care or nursing reminder teams
received influenza vaccine (1% and 6%, respectively).
During Year 3—after implementation of the electronic med-
ication administration record—among 805 patients dis-
charged by the pre-selected order team, most physicians
(n ? 665; 83%) accepted the pre-selected order, and there
was a dramatic increase in patient vaccination (n ? 286;
36%), Figure 1. Of 665 patients who had a physician’s order
for vaccination, 43% were vaccinated. Since approximately
30% of patients were vaccinated before hospitalization, we
estimate that 61% of eligible patients were vaccinated. Over
the three influenza seasons, there was a significant increase
in patient vaccination (Year 1 [0/36], Year 2 [8/66], Year 3
[286/805]; p ? 0.001), Figure 1.
Over time, physicians were increasingly likely to accept
pre-selected vaccination orders; by Year 3 83% of orders
were accepted. A bigger challenge was improving vaccine
administration after physician order, which improved dra-
matically after the medication administration record was
integrated into the electronic medical record. Also, since
most internal medicine patients met criteria for vaccination,
sophisticated rule-building was unnecessary. In fact, since
not all patients who meet high-priority criteria for vaccina-
tion could be identified using clinical data, building rules to
selectively trigger the CDS system would have resulted in
missed opportunities to vaccinate high-risk patients.
Our findings illustrate the tenet that to successfully imple-
ment CDS it is essential to address workflow integration,
healthcare worker-system interaction, local culture, and
transition of most processes to the electronic system.1,3,12–15
During Year 1, we attempted CDS in a predominantly
paper-based system; for example, electronic orders were
printed at the nurses station, which delayed notification of
nurses about the order during a time critical process. Since
our CDS rule was triggered by the “Discharge Patient”
order, nurses were required to vaccinate patients in the
relatively short time between the discharge order and the
patient’s departure. Although we considered other CDS
triggers, each potential solution had challenges. For exam-
ple, during Year 2 we tested automated electronic reminders
to nurses, triggered by patient admission. Unfortunately,
nurses rarely followed the policy. We considered using
temporal triggers, e.g., orders presented by hospital day, but
this was not an option with our CDS system. Finally, we
attempted to use the admission order as a trigger for the
pre-selected order, but could not resolve technical difficul-
During Year 1, physicians who were exposed to a reminder,
rather than the pre-selected order, did not place the order.
Likely because in part, there was no medication CPOE, and
physicians either had to write the order in the paper chart or
search for the electronic order at a time when there was no
CPOE for medications. During Year 2, increased vaccine
administration by nurses likely resulted from increased
functionality of the electronic medical record; for example,
availability of an electronic task list for nurses. In Year 2,
despite improved physician acceptance of the order and
increased vaccine administration, coverage levels remained
Since nursing and physician reminders were unsuccessful
during Years 1 and 2, in Year 3 we focused on using
F i g u r e 1.
by intervention and year.
Influenza vaccination rates of hospital patients,
Table 2 y Characteristics of Patients Evaluated during
the Cross-sectional Survey, 2003–2004
N ? 103%
Black or African American
Influenza vaccine history
Usually receives influenza vaccination
Received influenza vaccination this year
Reaction to influenza vaccination
Met criteria for vaccination
Internal medicine or family practice,
n ? 66
Surgery, n ? 37
Median age ? 52 years, inter-quartile range [43–62].
Gerard et al., Using Clinical Decision Support to Increase Flu Vaccination
pre-selected physician orders. To facilitate nursing adminis-
tration of vaccine during the discharge process, we intended
to test presenting the pre-selected order to physicians upon
both patient admission and discharge. However, despite
successful use of the rule in the test environment, we were
unable to reliably use the order triggered by patient admis-
sion. Specifically, influenza orders triggered by patient ad-
mission usually were not accepted by the system; therefore,
the pharmacist could not verify the order electronically.
Since Year 3 vaccination rates were dramatically improved,
we believe that the low vaccine administration rates during
Years 1 and 2 were primarily due to incomplete maturation
of the electronic system, rather than logistical challenges
posed by triggering the pre-selected order on patient dis-
Our experiences illustrate how CDS implementations re-
quire attention to local workflow, in particular, transition of
workflow from paper-based to electronic systems. This may
partly explain the relatively low penetration of CDS in
healthcare settings, especially since many institutions do not
have the expertise to tailor less flexible vendor-provided
systems to their needs.4,16Despite these challenges, we
achieved significant and meaningful increases in vaccination
coverage using CDS. By Year 3, over 50% of patients not
vaccinated before hospitalization, were vaccinated during
their hospital stay.
At our hospital, a written universally-applied standing orders
policy was ineffective, even after augmenting the policy with
electronic reminders to nurses. In addition to nurses’ concerns
about acting without an individual physician’s order, during
educational sessions many nurses expressed concerns about
the influenza vaccine. After recognizing these substantial local
barriers, during Year 3 we abandoned electronic nursing re-
minders—despite proven success at another institution.17
For the following reasons, we presented the pre-selected
order to all patients: incomplete electronic problem lists
compromise the sensitivity of electronic inferences of
chronic medical conditions, some criteria are not electroni-
cally captured (e.g., living with a high-risk person), we were
hesitant to increase nurses’ workload by creating an elec-
tronic form to screen patients, and influenza vaccination is
safe and effective for all patients.
Our findings are limited in that we evaluated patients in a
single, large, urban public hospital, nurses at other hospitals
may be more likely to respond to electronic reminders and
standing orders policies.17Also, we did not determine
previous vaccination history through patient interview;
however, a bedside interview may have influenced patients’
desire for vaccination and biased our results.
We encountered several challenges during implementation
of a CDS rule to increase influenza vaccination. These
challenges included local cultural issues—nurses were re-
luctant to carry out standing orders—and technical issues,
such as incomplete integration of our electronic medical
record and lack of functionality of a vendor-provided sys-
tem. Despite these challenges, we observed early and nearly
complete physician acceptance of the CDS-generated order.
After integration of the electronic medication administration
record, there was a dramatic increase in nurses’ administra-
tion of vaccine. Use of CDS can dramatically improve
patient care, but success may be realized only after under-
standing the local workflow and culture, and near-complete
transition from paper to electronic processes.
1. Gross PA, Bates DW. A pragmatic approach to implementing
best practices for clinical decision support systems in comput-
erized provider order entry systems. J Am Med Inform Assoc.
2. Kuperman GJ, Bobb A, Payne TH, et al. Medication-related
Clinical Decision Support in Computerized Provider Order
Entry Systems: A Review. J Am Med Inform Assoc. 2007;14:
3. Bates DW, Kuperman GJ, Wang S, et al. Ten Commandments
for effective clinical decision support: making the practice of
evidence-based medicine a reality. J Am Med Inform Assoc.
4. Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A,
Detmer DE. A Roadmap for National Action on Clinical Deci-
sion Support. J Am Med Inform Assoc. 2007;14(2):141–5.
5. CDC. National Center for Health Statistics. Electronic Medical
Record Use by Office Based Physicians: United States, 2005.
Available at: http://www.cdc.gov/nchs/products/pubs/pubd/
hestats/electronic/electronic.htm. Accessed June 30, 2007.
6. Jha AK, Ferris TG, Donelan K, et al. How Common Are
Electronic Health Records in the United States? A Summary of
the Evidence. Health Affairs 2006;25:w496–w507.
7. Bridges CB, Thompson WW, Meltzer MI, et al. Effectiveness and
cost-benefit of influenza vaccination of healthy working adults:
a randomized controlled trial. JAMA 2000;284:1655–63.
8. Gross PA, Hermogenes AW, Sacks HS, Lau J, Levandowski RA.
Efficacy of influenza vaccine in elderly persons: a meta-analysis
and review of the literature. Ann Intern Med. 1995;123:518–27.
9. Centers for Disease Control and Prevention. Influenza and
pneumococcal vaccination levels among persons aged ?65
years—United States, 2001. MMWR. 2002;51:1019–24.
10. Morita J. Addressing racial and ethnic disparities in adult
immunization, Chicago. J Public Health Manag Pract. 2006;12:
11. Centers for Disease Control and Prevention. Prevention and
control of influenza: Recommendations of the Advisory Com-
mittee on Immunization Practices. MMWR. 2003;52(RR08):1–36.
12. Kawamoto K, Houlihan CA, Balas A, Lobach DF. Improving
clinical practice using clinical decision support systems: a system-
atic review of trials to identify features critical to success. BMJ
2005;330:765(2April), doi:10.1136/bmj.38398.500764.8F (published
14 March 2005).
13. Kawamoto K, Lobach DF. Clinical Decision Support Provided
within Physician Order Entry Systems: A Systematic review of
Features Effective for Changing Clinician Behavior. AMIA
Annu Symp Proc. 2003;:361–5.
14. Miller RA, Waitman LR, Chen S, Rosenbloom ST. The anatomy
of decision support during inpatient care provider order entry
(CPOE): Empirical observations from a decade of CPOE expe-
rience at Vanderbilt. J Biomed Inform. 2005 December;38(6):
15. Nies J, Colombet I, Degoulet P, Duriex P. Determinants of
Success for Computerized Clinical Decision Support Systems
Integrated into CPOE Systemes: a Systematic Review. AMIA
Annu Symp Proc. 2006;:594–8.
16. Kaushal R, Shojania KG, Bates DW. Effects of Computerized
Physician Order Entry and Clinical Decision Support Systems
on Medication Safety. Arch Intern Med. 2003;163:1409–16.
17. Dexter PR, Perkins SM, Maharry KS, Jones K, McDonald CJ.
Inpatient Computer-Based Standing Orders vs Physician Re-
minders to Increase Influenza and Pneumococcal Vaccination
Rates. JAMA. 2004;292:2366–71.
Journal of the American Medical Informatics Association Volume 15Number 6 November / December 2008