Understanding the management of electronic test result notifications in the outpatient setting.

RESEARCH ARTICLE Open Access
Understanding the management of electronic
test result notifications in the outpatient setting
Sylvia J Hysong1*, Mona K Sawhney1, Lindsey Wilson1, Dean F Sittig2, Adol Esquivel3, Simran Singh4 and
Hardeep Singh1
Abstract
Background: Notifying clinicians about abnormal test results through electronic health record (EHR) -based “alert”
notifications may not always lead to timely follow-up of patients. We sought to understand barriers, facilitators, and
potential interventions for safe and effective management of abnormal test result delivery via electronic alerts.
Methods: We conducted a qualitative study consisting of six 6-8 member focus groups (N = 44) at two large,
geographically dispersed Veterans Affairs facilities. Participants included full-time primary care providers, and
personnel representing diagnostic services (radiology, laboratory) and information technology. We asked
participants to discuss barriers, facilitators, and suggestions for improving timely management and follow-up of
abnormal test result notifications and encouraged them to consider technological issues, as well as broader,
human-factor-related aspects of EHR use such as organizational, personnel, and workflow.
Results: Providers reported receiving a large number of alerts containing information unrelated to abnormal test
results, many of which were believed to be unnecessary. Some providers also reported lacking proficiency in use of
certain EHR features that would enable them to manage alerts more efficiently. Suggestions for improvement
included improving display and tracking processes for critical alerts in the EHR, redesigning clinical workflow, and
streamlining policies and procedures related to test result notification.
Conclusion: Providers perceive several challenges for fail-safe electronic communication and tracking of abnormal
test results. A multi-dimensional approach that addresses technology as well as the many non-technological factors
we elicited is essential to design interventions to reduce missed test results in EHRs.
Keywords: Decision Support Systems Clinical, Automated notification, diagnostic errors, abnormal diagnostic test
results, Medical Records Systems, Computerized, patient follow-up, patient safety, health information technology,
communication, primary care
Background
The American Recovery and Reinvestment Act of 2009
(ARRA) awards up to $63,750 of incentive payments to
health care providers who demonstrate “meaningful use”
of a “qualified electronic health record” (EHR), and will
eventually penalize providers who do not demonstrate
such meaningful use by 2015[1]. One important aspect
of the meaningful use concept is the application of
clinical decision support (CDS) tools to improve coordi-
nation and quality of health care[2]. For instance, real-
time electronic notification of abnormal test results via
the EHR may facilitate timely follow-up, particularly in
outpatient settings, where many results are not immedi-
ately life threatening and not verbally reported to order-
ing clinicians[3,4]. Outpatient test results are especially
vulnerable to “falling through the cracks” [5,6] and are
often cited as reasons for delays in diagnosis and treat-
ment, patient harm and malpractice claims[7-14].
To achieve meaningful use of EHRs as envisioned by
the federal government, providers need to be proficient
in use of the decision support features available in their
EHR and understand how they fit into the clinical
* Correspondence: sylvia.hysong@va.gov
1Houston VA Health Sciences Research & Development Center of Excellence,
The Center of Inquiry to Improve Outpatient Safety Through Effective
Electronic Communication, Michael E. DeBakey Veterans Affairs Medical
Center and the Section of Health Services Research, Department of
Medicine, Baylor College of Medicine, Houston, Texas, USA
Full list of author information is available at the end of the article
Hysong et al. BMC Medical Informatics and Decision Making 2011, 11:22
http://www.biomedcentral.com/1472-6947/11/22
© 2011 Hysong et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons
Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in
any medium, provided the original work is properly cited.

workflow. However, provider needs and the current
workflow are not always considered when designing
EHR systems[15]. In many EHR systems, an electronic
notification feature (e.g., the “View Alert” window used
by the U.S. Department of Veterans Affairs’ (VA) Com-
puterized Patient Records System (CPRS), or the “In-
Basket” feature available in Epic’s EpicCare EHR) deli-
vers test results to a message inbox that providers can
access after they login to the EHR. (see Additional file
1). Alerting through this asynchronous mechanism is
quite different from “synchronous” CDS alerts such as
drug-drug interaction (DDI) alerts, which interrupt
users when they are entering medication orders. While
synchronous alerting has been studied quite extensively,
[16-20] alerting through “asynchronous” channels has
received little attention. Unlike actions related to DDI
alerts, follow-up actions required of test results alerts
are not necessarily required immediately after alert
delivery. How these asynchronous alerts integrate into a
provider’s workflow is largely unknown.
We recently examined providers’ responses and fol-
low-up actions on over 2500 alerts of abnormal test
results in CPRS[5,6]. Of these, we found providers did
not acknowledge (i.e., did not read) 18.1% of alerts per-
taining to abnormal imaging results and 10.2% of abnor-
mal laboratory alerts. Furthermore, approximately 8% of
abnormal imaging and 7% of abnormal laboratory
results lacked timely follow-up at 30 days. We also
found that there was no significant relationship between
acknowledging an alert and timely follow-up. Thus,
despite delivery of test results directly to a clinician’s
View Alert window, abnormal results did not always
receive timely follow-up.
Clinicians do not optimally utilize all of the functions
in the EHR; for instance, we found that about half (46%)
of clinicians did not use the specific features of the
View Alert window that facilitate better processing of
electronic alerts[21]. Instead, providers often used hand-
written notes or external systems (e.g., ticklers on their
calendar) to help process their alerts[21]. Thus, many
factors beyond the technology itself will likely predict
how “meaningfully” providers will use CDS tools for test
result reporting in the future[22,23].
To obtain a comprehensive understanding of the man-
agement of test result alerts in EHRs and to explain why
abnormal results might be missed, we used a qualitative,
sociotechnical, systems-based approach. Our objective
was to conduct a qualitative study at two large VA facil-
ities to understand barriers, facilitators, and potential
interventions for effective and safe management of
abnormal test results delivered through the EHR. We
relied on human factors engineering principles to frame
our research questions and explore issues beyond the
confines of the computer.
Methods
Human Subjects
This study was approved by the Baylor College of Medi-
cine Institutional Review Board for compliance with
accepted human subject research practices consistent
with the Helsinki Declaration (Protocol # H-21817,
Improving Outpatient Safety Through Effective Electronic
Communication).
Design and Setting
We conducted three focus groups at each of two large,
geographically dispersed VA medical centers between
January and May 2009. Focus groups are ideally suited
for this type of research because they allow for live
interaction among participants, and richer data than
what survey methods could elicit[24]. Table 1 presents
basic characteristics for the two sites.
For almost a decade, both study sites have used the
Computerized Patient Record System (CPRS), the
EHR system in use at VA facilities nationwide. Within
CPRS, providers are notified of test results in a “View
Alert” window that is displayed when a provider logs
in. Although some functionality is configurable at the
facility and user levels, most software changes to
CPRS are made at the national level and disseminated
simultaneously to all VA facilities. Consequently,
CPRS configurations are far more standardized among
facilities than most other commercially available EHR
systems.
Table 1 Basic characteristics of participating sites and
focus group composition
Site Characteristics Site A Site B
Number of Patients enrolled 122,452 82,000
Outpatient visits/year 815,695 780,000
Academically affiliated? Yes Yes
Number of Primary Care Providers 38 16
Focus Group Composition
FG No. Role Site A Site B
1 Lab/Radiology Personnel 2 0
IT Personnel 1 1
Primary Care Providers 3 8
Specialist 1 0
2 Lab/Radiology Personnel 1 0
IT Personnel 1 0
Primary Care Providers 4 8
Specialist 0 0
3 Lab/Radiology Personnel 1 0
IT Personnel 1 0
Primary Care Providers 4 8
Specialist 0 0
Total 19 25
Hysong et al. BMC Medical Informatics and Decision Making 2011, 11:22
http://www.biomedcentral.com/1472-6947/11/22
Page 2 of 9

Participants and Sampling Frame
Forty-four full-time personnel representing primary care,
radiology, information technology (IT) and laboratory
services participated in the focus groups. We purpo-
sively selected participants from these fields specifically
because their job responsibilities involved considerable
interaction with test result notifications in the View
Alert window at different parts of the workflow. Addi-
tionally, the primary care providers (PCPs, which con-
sisted of physicians and physician assistants) were
purposively sampled based on previous analyses [5] for
having a high or a low number of alerts lost to follow-
up within a 30 day period. Groups were limited to the
recommended size of six to eight participants[24]. Table
1 presents the composition of each focus group by staff
specialty.
Procedure
Details of our data collection and analysis procedures
are published elsewhere [25] and are summarized here.
Participants were recruited via phone/email, and signed
an informed consent form before participating in the
focus groups after having the opportunity to ask ques-
tions about the study, including issues of anonymity and
confidentiality.
During the first two focus groups, we asked partici-
pants to discuss barriers and facilitators to successful
management and follow-up of abnormal test result
alerts and provide suggestions for improvement. We
encouraged participants to think beyond the CPRS user
interface, software, and hardware, and to consider
broader human factors engineering issues such as orga-
nizational, personnel, workflow, and work environment
concerns. Participants in the third focus group at each
site concurred or dissented with the most frequently
raised themes from the first two focus groups and dis-
cussed further barriers and suggestions for
improvement.
Data Analysis
We used thematic analysis [26] to analyze our focus
group transcripts, with the goal of identifying common
alert management barriers and facilitators and sugges-
tions for improving the alert system. Analysis tasks
included a) the development of an initial coding taxon-
omy; b) open coding, in which text passages were exam-
ined for recurring themes and ideas; and c) axial coding,
in which themes were organized into meaningful rela-
tionships. Figure 1 depicts the flow of analysis tasks.
Taxonomy Development and Open Coding
Two coders with qualitative research experience indepen-
dently coded the focus group transcripts for content per-
taining to barriers, facilitators, and suggestions for
improvement. The coded data sets were then merged and
reviewed by a third coder (the validator) to reconcile
nearly identical quotations, codes carrying different labels
yet referring to the same phenomenon, and codes need-
ing further discussion to reach consensus. The coding
team then met to review and reach agreement on discre-
pant codes and quotations. After a one-week waiting per-
iod to reduce priming effects, each coder independently
coded the finalized quotation list using the newly devel-
oped taxonomy. The validator again identified inter-
coder discrepancies, which were resolved by consensus.
Axial Coding
We first organized coded passages according to ground-
edness (i.e., the number of quotations to which a code
was assigned) to determine the most commonly cited
barriers, facilitators, and suggestions for improvement.
We then compared the patterns of coded passages by
site and used these comparisons to identify larger,
recurring themes.
Results
Figure 2 presents the barriers, facilitators, and sugges-
tions for improvement that were most frequently raised
by participants according to their groundedness; all
themes presented were mentioned by multiple partici-
pants. The most commonly cited barriers overlapped
considerably across sites and focus groups, despite dif-
ferences in site characteristics and focus group composi-
tion. Furthermore, these themes were raised by multiple
participants across the focus groups, suggesting they
were not simply an artifact of a single, dominant
participant.
Number of Alerts Received
The most frequently raised barrier was the number of
alerts received by providers. In addition to test result
Figure 1 Summary of coding process and analysis flow.
Hysong et al. BMC Medical Informatics and Decision Making 2011, 11:22
http://www.biomedcentral.com/1472-6947/11/22
Page 3 of 9

alerts providers received many other types of notifica-
tions, which complicated the task of reviewing test
results and providing timely follow-up care. Participants
expressed concern about both the total number of alerts
and the proportion of notifications perceived as unne-
cessary. Providers in all focus groups reported that their
already heavy clinical workloads left very little time for
the task of alert management:
“On an average it takes about two to three minutes
per alert. And we get sixty to seventy alerts per day.
So, there’s no time allowed for alerts... I’ve just fin-
ished seeing patients. I have to go back and handle
all the alerts. Some people actually come in on week-
ends. So, yeah, time is definitely a factor.” -PCP, Site
A
“One of the issues is just the sheer volume of alerts,
and there are a number of alerts that in all honesty
[you] really don’t have any business seeing.” -PCP,
site B
“I counted 150 alerts one day just to see how many
were coming in that normal day, and this is a fairly
regular day, 150 alerts. That’s a lot of time spent try-
ing to go through that while you’re seeing patients,
while there’s no in between time to get caught up.”
-PCP, Site B
Types of alerts perceived as unnecessary varied across
the sites, but at both sites providers discussed situations
when they were needlessly notified of events they
deemed as strictly “for your information.” These
included, for example, overly detailed status updates of
services performed outside primary care:
“The surgeon needs to take care of his own alerts. I
don’t need to be a backup for him. I mean, you
know, he’s licensed, right? He holds a license. He
needs to worry about his license. He needs to take
care of his stuff. And if every department did that, I
mean, that would cut down our workload by fifty
percent. That’s where the problem is, everybody
expects us to be the backup, and there’s really no
need.” PCP, Site A
“You could have half a dozen notifications on a given
consult which really are unimportant. The only thing
I really need to know about is if it was actually
scheduled and what the date was in case it’s some-
thing I want to have done soon, and this is way too
0 10 20 30 40 50 60 70
Alerts, in addition to daily clinic duties, contribute to a heavy workload
Providers perceive some types of alerts are unnecessary
Providers lack knowledge regarding features of the alert system in CPRS
Providers receive a high volume of alerts
Providers dislike getting alerts which are redundant
Problems relating to the communication between lab/radiology personnel and…
Providers don’t like when alerts disappear from the View Alert window
Providers lack sufficient training to manage alerts in CPRS
Providers have difficulty processing certain types of alerts in CPRS
Problems with hardware affects management of alerts
Providers feel the alerts should not disappear from the window
Generally, the alert system in CPRS is helpful
Providers like to receive important alerts for which they are not the ordering…
Ability to access quickly and electronically communicate test results
CPRS allows instant access to everyone else’s notes
CPRS allows users to customize their own alert notification preferences
CPRS has helpful features to manage and process alerts (e.g. sorting alerts)
Hardware has become faster
Ability to categorize and organize different types alerts
Bundle related alerts to reduce redundancy
Providers want the ability to keep or save alerts so they can view them later
Make improvements to the surrogate process
Providers want the ability to categorize and organize different types alerts
Only receive discharge summary alert for inpatients
Improve training on View Alert management in CPRS
Leave reminders for one’s self in CPRS for future action
Number of Mentions at each site
Th
em
e
Site A
Site B
B
a
r
r
i
e
r
s
Facilitators
Suggestions for Improvement
Figure 2 Most commonly cited barriers, facilitators, and suggestions for improvement, by site.
Hysong et al. BMC Medical Informatics and Decision Making 2011, 11:22
http://www.biomedcentral.com/1472-6947/11/22
Page 4 of 9

far away, or if it’s canceled altogether. I really don’t
care about any of the other notifications, all these
notes that they pass back and forth about whether or
not they contacted the patient, whether or not he
had transportation.” PCP, Site B
To further explore the problem of large numbers of
alerts, we conducted a co-occurrence analysis to exam-
ine common themes in participants’ proposed solutions.
We identified all passages in which suggestions for
improvement co-occurred with any of the three quan-
tity-related barriers most heavily discussed in the focus
groups: too many alerts, unnecessary alerts, and an
overly heavy patient-related workload created by the
alerts. As seen in Figure 3, the three barriers co-
occurred with a total of 17 suggestions for improve-
ment. Fourteen were associated with overcoming the
workload barrier; these suggestions involved both
changes to CPRS (e.g., ability to categorize alerts, bund-
ling alerts together) and changes to workflow (e.g., allo-
cating protected time to manage alerts). Ten of
seventeen suggestions applied to multiple barriers, sug-
gesting that these barriers are interrelated. Interestingly,
only three suggestions were uniquely associated with the
two barriers about number of alerts. This analysis sug-
gests that workload created by alerts is a complex bar-
rier needing multidimensional solutions.
Tracking and Categorizing Relevant Clinical Information
Another salient theme was providers’ desire for a
mechanism within CPRS to organize, track, and retrieve
alerts so that providers remember to follow up on
needed care. As the CPRS View Alerts system was
designed to alert providers so they could take action at
the time of the alert, no functions for longitudinal track-
ing currently exist in CPRS. Therefore, at both sites,
better EHR capabilities to help visualize, organize, and
track alerts ranked among the most frequently cited
suggestions.
I always wished to see a way to see or a way to know
because then I can know how they’re performing.
Figure 3 Co-occurrence analysis of participants’ proposed solutions to volume-related barriers.
Hysong et al. BMC Medical Informatics and Decision Making 2011, 11:22
http://www.biomedcentral.com/1472-6947/11/22
Page 5 of 9