Showing Your Work: Impact of annotating electronic prescriptions
with decision support results
Kevin B. Johnson*, Yun-Xian Ho, Cather Marie Cala, Coda Davison
Department of Biomedical Informatics, Vanderbilt University, 2209 Garland Ave., Room 428, Nashville, TN 37232, USA
a r t i c l e i n f o
Received 31 July 2009
Available online 6 December 2009
Clinical decision support
a b s t r a c t
e-Prescribing systems with decision support do not routinely communicate an adequate amount of infor-
mation regarding the prescribers’ decision to pharmacists. To address this communication gap in the e-
prescribing process, we implemented a system called Show Your Work (SYW) that appends alerts and
override comments to e-prescriptions generated by an e-prescribing system. To assess the quantitative
impact of this system, we conducted a randomized, double-blinded, controlled study to assess pharmacy
callback rates and types, and to uncover any unintended consequences of the annotations. Each day, SYW
output across the enterprise was turned ‘‘on” or ‘‘off” randomly for all e-prescriptions. A convenience
sample of three pharmacies, blinded to SYW status, submitted callback logs each day. These logs were
used to calculate the rate of and reason for callbacks. At the conclusion of the study, we surveyed the
50 most frequently used pharmacies in our area to assess the impact of SYW on satisfaction and commu-
A total of 202 callbacks had occurred yielding a callback rate of 45 callbacks/1000 prescriptions for SYW
‘‘on” days and 40 callbacks/1000 prescriptions for ‘‘off” days (p = 0.4). We received 38 surveys (76%
response rate) with 33 respondents commenting about SYW. Most respondents agreed (69%) that SYW
favorably impacted callbacks—especially with pediatric prescriptions (82%). Comments suggested that
SYW increased callbacks where necessary and decreased them in other situations, but did not contribute
to unnecessary callbacks. These findings support the continued and potentially expanded use of SYW by
e-prescribing systems to enhance communication with pharmacists.
? 2009 Elsevier Inc. All rights reserved.
e-Prescribing technology has become an important component
of the health care systems in ambulatory settings [1–6]. Groups
such as the eHealth Initiative  and the Institute of Medicine
 have promoted this technology, which is presently used by less
than 25% of practices nationally . However, this level of adoption
is likely to change, in part due to recent pay-for-performance pro-
grams provided by insurers such as the Center for Medicaid and
Medicare Services  and legislation passed as a part of the Amer-
ican Recovery and Reinvestment Act . Numerous studies note
both financial [1,10,11] and patient safety benefits [1,12,13] that
may be achievable through e-prescribing. Common areas of deci-
sion support include formulary and benefit checking, drug allergy
checking, drug–drug interaction, drug-condition checking, and
avoidance of drug side effects. While most e-prescribing systems
provide some of these warnings, these systems are not alone in
creating a safety net for patients receiving prescription medica-
tions. Many pharmacy information systems also include similar
warnings, and will result in a call to an ordering prescriber when-
ever a warning is provided to the pharmacist that may otherwise
have resulted in a change in the prescription by the prescriber
[14–16]. In most cases, pharmacists are only able to provide this
safety net when patients complete profiles that are entered into
pharmacy information systems—a practice that is inconsistently
Despite the provision of prescribing-related warnings in pre-
scribing systems, there is no current process by which warnings
that are triggered and overridden can automatically be conveyed
to a pharmacist completing a prescription order. Without this
information, pharmacists may have to call prescribers to under-
stand why a patient is being prescribed a medication to which he
or she is allergic, or may be unable to verify a dose calculation in
a pediatric patient. With the use of electronic prescription writing
tools, there is no limit to the information that can automatically be
added to a prescription to convey decisions made during the pre-
scribing process. At our institution, we have constructed a method
to annotate prescriptions automatically, called ‘‘Show Your Work”
(SYW). This process adds notes below each medication order to de-
scribe any decision support warnings that were displayed at the
time of prescribing, any overrides to drug alerts provided by the
1532-0464/$ - see front matter ? 2009 Elsevier Inc. All rights reserved.
* Corresponding author. Fax: +1 615 936 1427.
E-mail address: Kevin.email@example.com (K.B. Johnson).
Journal of Biomedical Informatics 43 (2010) 321–325
Contents lists available at ScienceDirect
Journal of Biomedical Informatics
journal homepage: www.elsevier.com/locate/yjbin
prescriber during the session, and any dose calculations for pediat-
ric prescriptions. Show Your Work also notes when specific tasks
were done manually or could not be accomplished. Fig. 1 shows
an example of SYW on an actual prescription, and Table 1 shows
a series of SYW output types. This system has been implemented
for approximately 2 years, and is in use throughout our e-prescrib-
While the provision of annotations on prescriptions may seem
risk-free, there are numerous potential ways that it could be detri-
mental. First, these annotations could be falsely reassuring, espe-
cially if providers do not actually take into account the
information that is communicated to pharmacists. Also, annota-
tions could be confusing to pharmacists and could generate call-
backs based on information that is not clearly presented. The
goal of this project was to explore the impact of SYW on callbacks
and on perceived changes to pharmacy workflow.
The study utilized RxStar, an e-prescribing system installed in
the ambulatory clinics under a single clinical practice group
throughout three counties in middle Tennessee. RxStar is a web-
based e-prescribing system that is fully integrated into an EHR
and is used by over 2000 prescribers. It supports electronic faxing
of prescriptions, and provides warnings for over/under dosing,
drug interactions, formulary/benefit management, age/condition-
specific warnings, as well as pediatric dose calculation. RxStar
has been in place for over 5 years, and in use in all medical special-
ties. During the time of the study, RxStar prescriptions were either
printed or electronically faxed to pharmacies throughout Tennes-
see and surrounding states, and comprised approximately 35% of
prescriptions received by three pharmacies on the Vanderbilt
2.2. Study design
The research team conducted two evaluations. The first evalua-
tion evaluated the impact of SYW on the rate of and reasons for
callbacks, while the second evaluation explored the impression of
SYW in local pharmacies.
To test the hypothesis that SYW impacted the rate of and rea-
sons for callbacks, we conducted a randomized, controlled, dou-
ble-blinded study of callbacks with and without the addition of
SYW to e-prescriptions. During this evaluation, all prescriptions
generated by RxStar users throughout all ambulatory sites had
SYW annotations either hidden (SYW ‘‘off”) or displayed (SYW
‘‘on”) when transmitted to pharmacies. We used block randomiza-
tion with a block size of 5 to guarantee that each week would have
a roughly equal number of both ‘‘on” and ‘‘off” days, and mini-
mized potential confounding associated with the predictable
scheduling of clinicians in academic clinic settings. This evaluation
occurred from April 4, 2007 until August 3, 2007 (119 days), with a
total of 57 days where SYW was on (SYW on), and 62 days where
SYW was off (SYW off). Providers were unaware of the true state
of SYW, which always appeared ‘‘on” for providers during the dura-
tion of the study. Pharmacists, who generally do not receive anno-
tations on prescriptions, and for whom Vanderbilt e-prescriptions
were probably a minority of received prescriptions, did not know
whether annotations were simply omitted on ‘‘off” days, or not
applicable for a given prescription. Because pharmacists did not re-
ceive annotations on prescriptions from other sites, and the benefit
of this additional communication was unknown, we were comfort-
able with the notion that pharmacists would use conventional
methods (i.e., talking with families and communication as needed
The callback study focused on the three pharmacies affiliated
with our academic center. These pharmacies rely solely on their
pharmacy information system and information on the prescription
for any decision support. They do not use the local electronic
health record, and they do not have access to the e-prescribing sys-
tem. Before beginning the study, we developed a pharmacy call
back log form that allowed pharmacists to state the date and time
for each callback, the reason for of each callback, and the type of
prescription generating the callback. We worked closely with phar-
macists to develop categories describing the types of prescriptions
received, the reasons for callbacks, and the overall structure of the
form to integrate it into their existing callback workflow. After
testing and an initial analysis of pilot survey responses from phar-
macists, we were identified six categories for callbacks, as shown
in Table 2.
Pharmacists working in these pharmacies used this form to doc-
ument all callbacks beginning two weeks before the study, and re-
ceived periodic incentivesto
throughout the study period.
In the second evaluation we collected information about phar-
macists’ perceptions of the SYW system impact on callbacks,
checking for errors and checking for insurance eligibility. We
Fig. 1. Example of Show Your Work annotations below an electronically generated prescription.
Example Show Your Work output types.
Free text medication
‘‘Potential allergy: overridden by prescriber. Reason provided—‘‘No other option”
‘‘Dose warning: total dose amount of 2500 MCG/day exceeds the dosing range of 50–500 MCG/day”
‘‘Dose calculated based on weight of 23 kg and formula of 0.8 mg/kg/day”
‘‘Formulary status: not reimbursed”
‘‘Dose is manually calculated”
‘‘Free text medication: no automated decision support has been applied to this medication. Please verify it for accuracy”
K.B. Johnson et al./Journal of Biomedical Informatics 43 (2010) 321–325
developed a 7-item survey and tested it for face validity before dis-
tributing it to the 50 pharmacies receiving the highest volume of e-
prescriptions from Vanderbilt prescribers, as derived from an anal-
ysis of our prescription faxing logs over the previous year. Respon-
dents rated statements like ‘‘SYW helps me avoid callbacks” and
‘‘SYW helps me check for potential errors” on a 5-point Likert scale.
The survey also included free-text comment boxes to allow phar-
macists to provide explanations for their responses. Each phar-
macy director completed a survey for that pharmacy. To
maximize response rate, surveys were distributed to pharmacies
using each pharmacy’s preferred method (via fax, e-mail, or postal
service). An initial question asked respondents if they were famil-
iar with the SYW feature, and if not, directed them to send in the
survey without completing it.
2.3. Data analysis
Callback log results were analyzed using R . The rate of call-
backs was analyzed using a chi-squared test, as were the relative
frequency of callbacks in each prescription type and for each call-
Pharmacy perception results were entered into SurveyMonkey
(www.surveymonkey.com) for descriptive analysis. We analyzed
Likert questions descriptively using R. Comments about SYW were
reviewed by one investigator and summarized into themes. Each
comment was associated with exactly one theme.
Table 3 summarizes the total RxStar-generated prescriptions
during the study period, along with the callbacks generated as a
percentage all RxStar prescriptions for each arm of the study.
Fig. 2 summarizes the types and frequencies of SYW messages re-
ceived by pharmacies during the second study period. Table 4
shows the difference between pharmacy callbacks rates for RxStar
prescriptions generated with SYW off and on, respectively. There
Reasons for callbacks based on pilot study.
Need to change medication-not covered, prior authorization needed, therapeutic duplication, wrong duration-should be 90 days, early refill, quantity, multiple
patients on one prescription
Illegible prescription, incorrect strength or concentration, bad range-order, too much/too little to dispense, dispense not in words, signature not legible, name not
printed on rx, no physician signature, address not on file, dea not on file, date missing, multiple controlled substance meds, patient name missing, patient date of
Inappropriate medication for indication, patient instructions contrary to rx directions, incorrect dosing, drug–drug interaction, missing duration, wrong medication,
unclear instructions, incorrect frequency, incorrect route
Medication out of stock or not stocked
Patient desires alternate
Change in callback rate with and without SYW.
SYW off N (%)SYW on N (%)
Fig. 2. SYW annotation categories and frequencies. Formulary compliance annotations appeared on all prescriptions for which insurance information was available (>60,000
per month), and are not shown here.
Callback reasons with and without Show Your Work on.
Callback reasonSYW off (%)SYW on (%)
Chi-sq(4) = 2.388, p = 0.665 (Fisher’s exact test, p ? 1).
K.B. Johnson et al./Journal of Biomedical Informatics 43 (2010) 321–325
was no significant difference in the callback rates between the two
periods (0.4% vs 0.45%; p = 0.47) or in the high-level categories of
SYW perception survey results are summarized in Table 5. A to-
tal of 38 out of 50 high-volume pharmacies responded (76% re-
sponse rate) with 5 pharmacy directors unable to recall if they
had noticed SYW at the bottom of a prescription. Therefore, re-
sponses were evaluated for the remaining 33 pharmacies that re-
sponded and were able to recall seeing SYW annotations.
When asked if SYW helped avoid callbacks (Table 3, Question 1)
the majority of respondents agreed or strongly agreed (69%). Phar-
macists found the allergy override information helpful (Table 3,
Question 6, 69% agree or strongly agree). A majority of pharmacists
(79%) felt that information about patient’s insurance eligibility was
less helpful; 41% of pharmacists were neutral, 31% were in dis-
agreement, and 7% were in strong disagreement with the state-
ment ‘‘SYW was helpful with insurance eligibility” (Table 3,
Comments about the SYW functionality are summarized as
themes presented in Table 6. Out of six themes extracted from
all comments, the most comment theme related to improved abil-
ity to verify pediatric doses (16 out of 36 comments). For example,
one respondent stated, ‘‘Date of birth is the usual info we get from
parents or on the Rx when it is received, but this is less accurate in
terms of dosing accuracy and the same age child can vary widely in
regards to weight.”
Two respondents noted that SYW increased callbacks—in both
cases, this behavior was related to potential allergies. One respon-
dent commenting about the allergy alert noted that SYW prompted
a call to the prescriber, ‘‘b/c sometimes an allergy was listed by a
medand the medicationwas
Two comments suggested additional information to add to the
annotations. For example, because SYW does not list the actual
reaction when noting an allergy, a respondent stated, ‘‘[I] will still
call on allergies after consulting patient if they are unsure of reac-
tion.” Another respondent, when discussing the value of SYW,
noted that ‘‘It is not helpful with insurance eligibility for us be-
cause it does not include enough information regarding Rx group#
and ID#.” Five comments requested new features, including the
addition of rounded pediatric doses , a comprehensive patient
medication list on the prescription , and a patient diagnosis
(which is already an option but rarely entered by the prescriber)
. Only three comments specifically noted examples where
SYW decreased callbacks.
Implementation of e-prescribing affords numerous opportuni-
ties to impact patient’s safety [7,14,18–23]. This study was the first
of its kind to examine the incorporation of a prescription annota-
tions tool in an e-prescribing system, and complements research
done by Rupp to understand best practice recommendations in
chain pharmacies . Results suggest that this easy to implement
e-intervention impacts the communication between providers and
pharmacists. Although the quantitatively reported rate of phar-
macy callbacks was not significantly different between our SYW
and control groups, qualitative review of pharmacist comments
suggests a potential positive impact of SYW on safety by clarifying
how the prescription was generated. In some cases, such as with
pediatric dosing, pharmacists report that this knowledge allows
them to verify the accuracy of doses and to look for calculation er-
rors. In other cases, such as with allergy alerts, they report that it
may cause them to call the provider or to talk with the patient to
better understand the nature of the reaction. In no cases did it re-
sult in harm to the patient, the provider or the pharmacy workflow.
Although qualitative data presented here were in support of the
continued exploration of SYW functionality in e-prescribing sys-
tems, we found no significant change in the callback rates in our
randomized trial. There are several factors that may account for
this. The study utilized a self-report method to detect changes in
pharmacy callbacks and may well have been limited by collection
challenges during the second phase of the study. Of particular note,
one of the pharmacies changed its physical location during the
trial. Although, the randomized design should have impacted both
the control and intervention days equally, it is possible that call-
backs were largely underreported during the relocation making
any difference between the callback rates appear negligible due
to low numbers.
The study also was limited to quantitative callback data ob-
tained only from pharmacies within close proximity to the investi-
gators. This non-random selection of sites may have introduced
selection bias if these pharmacies do not represent the community.
A larger, more random and sustained sampling of pharmacies may
be necessary to reveal any noticeable difference in callback rates as
a result of implementing the SYW functionality.
Nonetheless, qualitative comments offered by other pharmacies
in our area suggest that SYW may not necessarily affect the volume
of callbacks, although it may impact the quality of callbacks. These
pharmacies provided specific examples of when SYW generated
callbacks, as well as situations where it prevented them. It appears
that SYW helped pharmacists communicate more clearly with pre-
scribers. The availability of SYW information on the prescription
allows pharmacists to make a more well-informed callback which
can thereby ensure safer medication dispensing and expedite the
prescription filling process.
Show Your Work pharmacists perception survey results (n = 33).
Strongly disagree (%)Disagree (%) Neutral (%)Agree (%) Strongly agree (%)
1. Helped me avoid callbacks
2. Helped me check for potential errors
3. Caused me to call the prescriber back
4. Was helpful in pediatric cases
5. Was helpful with insurance eligibility
6. Was helpful with avoiding callbacks due to patient-reported allergies
7. Was helpful avoiding callbacks due to low or high doses
Show Your Work pharmacists perception survey comment themes.
Improving communication between prescribers and dispensers
Decreases callbacks in some cases
Pediatric dosing information helps check for potential errors
Increases callbacks in some cases
Need more information to be included in annotations
New Show Your Work feature request
K.B. Johnson et al./Journal of Biomedical Informatics 43 (2010) 321–325
The study focused on one e-prescribing system whose prescrip- Download full-text
tion output may have impacted pharmacist perceptions about the
value of SYW. However, the results appear to provide useful data
about the potential uses of prescriptions annotations and did not
appear confounded by the use of only one e-prescribing environ-
ment. We found that an SYW system can improve communication
with pharmacists, but additional studies are necessary to more di-
rectly establish a relationship between prescription annotations
and the types of callbacks that may be motivated by or mitigated
by this technology.
This study examined the incorporation of a prescription annota-
tions tool called Show Your Work, SYW, in an e-prescribing system.
We found no significant change in callbacks, although results sug-
gest that SYW may alter the rate at which specific types of call-
backs occur. This study is a fundamental step in the development
of a SYW system that can improve pharmacy workflow, perceived
quality of care, and foster more effective partnering between phar-
macists and prescribers.
This project was funded by AHRQ R03-HS016261. The authors
appreciate the efforts of Cara Baughman for assistance with data
collection and analysis and S. Trent Rosenbloom for carefully
reviewing this manuscript.
 Teich JM, Marchibroda JM. Electronic prescribing: toward maximum value and
rapid adoption. Washington, DC: eHealth Initiative; 2004 [April 14, 2004
Contract No.: Document Number].
 Grossman JM, Gerland A, Reed MC, Fahlman C. Physicians’ experiences using
 Joch A. Rx for safer ambulatory care. The adoption rate is low but potential is
high for e-prescribing. Health Commun Inform 2003;20(11):60.
 Kaufmann MD. Focus on: information technology. Electronic prescribing: an
update. J Drugs Dermatol 2005;4(1):106–7.
 Mayer T. E-prescribing hits the medical world: what physicians need to know
to get the right technology. J Med Pract Manage 2001;17(2):103–5.
 Friedman MA, Schueth A, Bell DS. Interoperable electronic prescribing in the
 Aspden P. Institute of Medicine (U.S.). Committee on identifying and
preventing medication errors. Preventing medication errors. Washington,
DC: National Academies Press; 2007.
 Services CfMaM. Electronic prescribing incentive fact sheet; 2008 [updated
2008; cited 2009 3/13/2009]. Available from: http://www.cms.hhs.gov/pqri/
 OpenCongress (Sunlight Foundation). American Recovery and Reinvestment
Act of 2009; 2009 [updated 2009; cited 2009 3/13/2009]. Available from:
 Linder JA, Ma J, Bates DW, Middleton B, Stafford RS. Electronic health record
use and the quality of ambulatory care in the United States. Arch Intern Med
 Ross SM, Papshev D, Murphy EL, Sternberg DJ, Taylor J, Barg R. Effects of
electronic prescribing on formulary compliance and generic drug utilization in
the ambulatory care setting: a retrospective analysis of administrative claims
data. J Manag Care Pharm 2005;11(5):410–5.
 Gurwitz JH, Field TS, Harrold LR, Rothschild J, Debellis K, Seger AC, et al.
Incidence and preventability of adverse drug events among older persons in
the ambulatory setting. JAMA 2003;289(9):1107–16.
 Corley ST. Electronic prescribing: a review of costs and benefits. Top Health Inf
 Wong IC, Wong LY, Cranswick NE. Minimising medication errors in children.
Arch Dis Child 2009;94(2):161–4.
on electronic prescriptions. J Am Pharm Assoc 2009;49(1):59–64.
 Cotter SM. Avoiding drug errors. Pharmacists and computerised prescribing
can help. BMJ (Clin Res Ed) 1995;311(7016):1367–8.
 R Development Core Team. R: a language and environment for statistical
computing. Computing RFfS, editor. Vienna, Austria; 2005.
 Bryan CS. Strategies to improve antibiotic use. Infect Dis Clin North Am
 Moore J. Electronic prescribing—‘the perfect prescription’. BJHC&IM (Br J
Healthcare Comput Inf Manage) 1996;13(4):25–7.
 Lambert BL. Predicting look-alike and sound-alike medication errors. Am J
Health Syst Pharm 1997;54(10):1161–71.
 Cohen HG. Medication errors—a system problem. Todays Surg Nurse
 Goldspiel BR, DeChristoforo R, Daniels CE. A continuous-improvement
approach for reducing the number of chemotherapy-related medication
errors. Am J Health Syst Pharm 2000;57(Suppl. 4):S4–9.
 Goldstein MK, Hoffman BB, Coleman RW, Tu SW, Shankar RD, O’Connor M,
et al. Patient safety in guideline-based decision support for hypertension
management: ATHENA DSS. Proc AMIA Symp 2001:214–8.
 Rupp MT, Warholak TL. Evaluation of e-prescribing in chain community
pharmacy: best-practice recommendations.
J Am Pharm Assoc
K.B. Johnson et al./Journal of Biomedical Informatics 43 (2010) 321–325