Comparison of computerized surveillance and manual
chart review for adverse events
Aldo Tinoco,1R Scott Evans,1,2Catherine J Staes,1James F Lloyd,2
Jeffrey M Rothschild,3Peter J Haug1,2
Objective To understand how the source of information
affects different adverse event (AE) surveillance methods.
Design Retrospective analysis of inpatient adverse drug
events (ADEs) and hospital-associated infections (HAIs)
detected by either a computerized surveillance system
(CSS) or manual chart review (MCR).
Measurement Descriptive analysis of events detected
using the two methods by type of AE, type of information
about the AE, and sources of the information.
Results CSS detected more HAIs than MCR (92% vs
34%); however, a similar number of ADEs was detected
by both systems (52% vs 51%). The agreement between
systems was greater for HAIs than ADEs (26% vs 3%).
The CSS missed events that did not have information in
coded format or that were described only in physician
narratives. The MCR detected events missed by CSS
using information in physician narratives. Discharge
summaries were more likely to contain information about
AEs than any other type of physician narrative, followed
by emergency department reports for HAIs and general
consult notes for ADEs. Some ADEs found by MCR were
detected by CSS but not verified by a clinician.
Limitations Inability to distinguish between CSS false
positives and suspected AEs for cases in which the
clinician did not document their assessment in the CSS.
Conclusion The effect that information source has on
different surveillance methods depends on the type of AE.
Integrating information from physician narratives with CSS
using natural language processing would improve the
detection of ADEs more than HAIs.
pressure to demonstrate the quality of care they
deliver.1e6This pressure originates from healthcare
consumers, state and national legislation, health-
care accreditation and credentialing agencies, and
quality improvement organizations. Healthcare
organizations generally rely on manual chart
review (MCR) to retrospectively measure quality
and safety. Yet this ‘gold standard’ is too time-
intensive and costly to be the sole means of
routinely identifying patient events of interest.7To
improve the efficiency of quality monitoring,
hospitals have implemented computerized surveil-
lance systems (CSS).8e14Through automation,
some hospitals have replaced retrospective, passive
monitoring with prospective, active surveillance that
allows concurrent interventions and improvement
in quality and safety.
of information to detect events. Computerized
surveillance system relies on clinical data that is
numeric or coded, such as pharmacy orders, labo-
ratory results, and claims data. Yet, prior studies
have shown that physician narratives and nursing
notes contain information about adverse events
(AE) not found elsewhere in the patient record.15
CSS that can access both coded and freetext datad
such as that found in unstructured narrativesdmay
cost associated with MCR alone.
Prior studies on CSS that use information in
physician narratives have focused on specific types
of documents, such as discharge summaries.16e18
Discharge summaries are valuable for retrospective,
post-discharge measurement of AEs. However, to
provide timely notification to providers and patient
safety personnel, prospective surveillance for AEs
requires real-time access to information throughout
the admission. Thus, we wanted to evaluate the
utility of other types of physician narratives
commonly found electronically in the inpatient
The CSS developers need to know what infor-
mation about AEs is relevant, where to look for it,
how it is represented, and how to extract it from
documents. Since MCR utilizes information from
both narrative and non-narrative sources, we
assumed that MCR would detect some cases
missed by CSS. To investigate this, we designed
a study to: (a) compare the type of AEs detected by
either MCR or CSS, (b) assess the features
associated with events found only by MCR, and (c)
identify opportunities for improving event detec-
tion by computerized surveillance. From a sample
of inpatient admissions during a specific time
period, we identified AEs that were detected only
by MCR, only by CSS, or by both methods. To aid
efforts to improve CSS, we collected actual phrases
from all electronic physician narratives throughout
a hospital admission that contained information
about adverse drug events (ADE) and hospital-
associated infections (HAI).
The study was performed at LDS Hospital, a major
teaching hospital in Salt Lake City, Utah. The
patient record at LDS Hospital has both electronic
and paper-based components. The HELP (Health
Evaluation through Logical Processing) system has
been operational at LDS Hospital since 1972 and
manages both clinical and financial patient infor-
mation.19In addition to billing and administrative
codes for each hospital admission, this electronic
system manages information from several clinical
<Additional appendices are
published online only. To view
these files please visit the
journal online (www.jamia.org).
1Department of Biomedical
Informatics, University of Utah,
Salt Lake City, Utah, USA
Intermountain Healthcare, Salt
Lake City, Utah, USA
3Division of General Medicine
and Primary Care, Department
of Medicine, Brigham and
Women’s Hospital, Boston,
Dr Aldo Tinoco, Department of
University of Utah, 26 South
2000 East, Room 5775 HSEB,
Salt Lake City, UT 84112, USA;
Received 15 February 2011
Accepted 23 April 2011
J Am Med Inform Assoc 2011;18:491e497. doi:10.1136/amiajnl-2011-000187491
Research and applications
domains: admission, discharge, and transfer (ADT)/registration,
pharmacy, laboratory, microbiology, nurse charting, and physi-
cian narratives, etc. The following physician narratives are
dictated, transcribed, and stored in the HELP system as freetext
physical report, consultant note (including bedside procedures),
However, daily inpatient progress notes are handwritten, paper-
based documents. Other paper-based parts of the patient record
include intraoperative physician orders and anesthesiology notes.
Printouts from the electronic record and the paper-based content
of the patient record are stored together as a hardcopy chart.
Computerized surveillance system
The HELP system has an integrated CSS that prospectively
screens electronic patient data for indicators of AEs, including
HAIs and ADEs. The HAI detection criteria used by CSS were
originally based on the guidelines from the Study of the Efficacy
of Nosocomial Infection Control and the Centers for Disease
Control and Prevention (CDC).8 20e22Using these criteria, CSS
evaluates each patient’s ADT, microbiology, serology, radiology,
and surgery data for evidence of an HAI. In addition to routine
HAI surveillance, daily urine samples from all catheterized
patients were obtained as part of an existing, hospital-wide
urinary catheter surveillance program.23The ADE detection
criteria used by CSS include various clinical triggers such as
antidote orders, laboratory test orders, abnormal laboratory test
results and vital signs.9 24Suspected cases are flagged by CSS and
reported to surveillance personnel for validation. An infection
preventionist (previously ‘infection control practitioner’) or
a clinical pharmacist verifies each HAI or ADE, respectively,
using information from the patient record, direct bedside
observations, and interviews with patients and their providers.
All suspected and verified cases are stored in the CSS database.
A sample of inpatient admissions to LDS Hospital had been
selected earlier for a comprehensive, multi-institutional research
investigation of AEs (‘workload study’).25The approach used in
the workload study to prescreen hospital admissions for possible
AEs was originally developed during the Harvard Medical Prac-
tice Study and refined in subsequent studies. This method for
prescreening possible AEs used a set of diagnostic and procedure
codes associated with a higher-than-average likelihood of an
AE.26 27In the workload study, admissions to the medical-
surgical services of LDS Hospital between October 1, 2000 and
December 31, 2001 were screened, which resulted in a sample of
2137 unique, prescreened patient admissions. Manual chart
review by trained chart abstractors was used to identify AEs
including HAIs and ADEs. In addition to inpatient events, we
included events detected by MCR that spanned multiple
admissions and events that led to new admissions to LDS
Hospital (eg, a postoperative wound infection presenting several
days or weeks after discharge for a surgical procedure).
We built a gold standard cohort of AEs by aggregating the AEs
detected by either CSS or MCR from the study sample of
admissions. We compared the AEs missed prospectively by CSS
with those detected retrospectively by MCR from this study
cohort, payingparticular attention tothe information about each
event contained in physician narratives.We sought to understand
what information CSS would need in order to detect events it
missed, the type of physician narrative in which the information
was found, and how it was represented in the narrative.
Case finding by method
Each detection method used formal case definitions of HAIs and
ADEs. For HAIs, both MCR and CSS used the CDC’s National
Nosocomial Infections Surveillance System surveillance case
definitions: infections acquired during hospital care that were
not present or incubating at admission.28 29Both methods used
CDC detection criteria to describe and verify the following types
of HAIs: bloodstream infections (BSIs), lower respiratory tract
infections (LRTIs), surgical site infections (SSIs), and urinary
tract infections (UTIs). For ADEs, MCR used the Institute of
Medicine’s case definition: ‘an injury to the patient resulting
from medical intervention related to a drug.’27
resulting from use of fluids and blood products were also
included. The CSS used the following ADE case definition: ‘a
response to a drug that is noxious and unintended and occurs at
doses normally used in man for the prophylaxis, diagnosis, or
therapy of disease, or for modification of physiological func-
tion.’9The definitions of ADEs used by MCR and CSS require
a causal association between the drug and manifestations related
to the action of the drug. During the workload study, infor-
mation was abstracted from the patient chart that fulfilled
surveillance criteria for each HAI and ADE.
We matched the HAIs and ADEs identified retrospectively by
MCR from the workload study sample with those identified
prospectively using CSS. For HAIs, an event was considered to
be a match between methods when the event occurred at the
same anatomic site during the same hospital admission. For
ADEs, an event was considered a match between methods when
the ADE was attributed to the same causative drug or fluid
during the same hospital admission. Only the first occurrence of
an HAI or ADE per admission was counted; thus, recurrences of
the same AE for the same patient during a single inpatient
admission were not included.
Review of events only identified using MCR
The phrases contained in physician narratives used to identify
AEs were the primary outcome of interest. One investigator
(AT) reviewed the information collected during the workload
study about each HAI and ADE identified only by MCR. All
electronic physician narratives for the corresponding inpatient
admissions were reviewed. Handwritten, inpatient progress
notes were not included in this study because their content is
not currently stored in the electronic medical record and would
not be accessible to the CSS. Each phrase related to an AE from
an electronic physician narrative and the corresponding type of
narrative was recorded in a study database. All phrases collected
are available in a separate appendix available as an online data
supplement (www.jamia.org). Specific event attributes for HAIs
and ADEs were used to classify each phrase (table 1). Investi-
gators matched the data from each phrase about HAIs to CDC
surveillance criteria, which were grouped into the following
categories: manifestation, intervention, response to treatment,
and assessment. For example, the phrase ‘The patient developed
wound cellulitis of the right inguinal lymphadenectomy and
was begun on Keflex and received 1 day of IV antibiotics of
cefazolin and was improving at time of discharge’ was catego-
rized in the following fashion:
MANIFESTATION: Abnormal sign/symptom¼wound cellulitis
INTERVENTION: Anti-infective treatment¼cefalexin
INTERVENTION: Anti-infective treatment¼cefazolin
RESPONSE: Improved manifestation¼resolving cellulitis.
At the time of the study, no formal event criteria were
available for ADEs. For each confirmed ADE case, investigators
matched the data in each text phrase with one of the following
492J Am Med Inform Assoc 2011;18:491e497. doi:10.1136/amiajnl-2011-000187
Research and applications
general event attributes of an ADE: drug, manifestation, inter-
vention, response to treatment, and assessment by a physician
(eg, recognition of an ADE) (table 1). For example, the phrase
‘pt. developed significant angioedema to tongue felt due to the
ACE inhibitor and this med had to be stopped’ was categorized
in the following manner:
DRUG: Therapeutic category¼ACE inhibitor
MANIFESTATION: Abnormal sign/symptom¼angioedema
INTERVENTION: Stop medication¼ACE inhibitor
Data analysis and management
We calculated the proportion of AEs in the sample detected by
each surveillance method and classified each event by AE type,
AE attributes, and sources of the information. Unique, mean-
ingless identifiers were assigned to each AE, the corresponding
patient, and the hospital admissions involved with each AE.
Information about each case, its abstracted text phrases, and the
source of information were stored in a password-protected
database application built using Microsoft Access 2003 and
Microsoft Visual Basic 6. This project was approved by the
institutional review boards of the University of Utah and LDS
Type of AEs detected using CSS and MCR
The distribution of AEs detected by MCR and CSS differed
between HAIs and ADEs. As shown in table 2, CSS detected
more than twice as many HAIs from the same sample of
admissions as MCR. For ADEs, similar numbers of events were
detected by CSS and MCR. However, the overlap for ADEs
detected by both methods was smaller than the overlap for HAIs
(3% vs 26%).
We identified reasons why CSS had missed the cases that were
only identified by MCR. All BSIs in the sample were detected by
CSS, since CSS detected the microbiological evidence of
each BSI. Several SSIs identified only by MCR had positive
microbiology culture results that could have been used by CSS’s
logic. However, the specimen sources assigned to those culture
results were freetext values that CSS had not been programmed
to interpret. The remaining SSIs did not have microbiological
evidence of an infection. The LRTIs and UTI identified only
by MCR also did not have microbiological evidence. The MCR
detected those HAIs using information contained only in
the physician narratives, such as anti-infective treatments,
radiographic findings, and physician diagnoses of an infection.
For ADEs, we identified multiple reasons why CSS had not
detected events that were found using MCR. The ADEs were
not detected by CSS when their event triggers were not used in
the CSS logic (eg, patient symptoms and physician assess-
ments). For example, all ADEs caused by narcotic analgesics
manifested as mental status changes. The CSS missed other
ADEs when a trigger was not available in the expected coded
form or data source. For instance, rescue medications adminis-
tered to a patient during cardiopulmonary arrest or during
a surgical procedure may have been recorded on a pre-printed
paper form but not the electronic medication administration
record. In some cases, CSS did not generate an alert as expected
even though the electronic signal was available, such as an
electronic medication order for vitamin K. Finally, some ADEs
were missed by CSS when a suspected case had been flagged by
CSS but not verified by a clinical pharmacist at LDS Hospital.
Type of information about AEs detected by MCR but missed by
Since all BSIs in the sample were detected by CSS, no infor-
mation from electronic physician narratives was collected for
this type of HAI. All other HAIs and ADEs detected only by
MCR had at least one event attribute found in a physician
narrative (table 3). For all event attributes, a greater percentage
of HAIs than ADEs had information contained in an electronic
physician narrative. The type of event trigger most likely to be
found in an electronic physician narrative was an intervention for
HAIs (97%) and a manifestation for ADEs (52%). The event
attribute least likely to be found for both HAIs and ADEs was
Attributes used to categorize the information contained in physician narratives about ADE or HAI events
AttributeDescription ADE example HAI example
Causative drugThe actual or missed administration of
a drug, fluid, or biological product that
was stated to have a causal association
with an ADE manifestation or is the target
of an intervention to treat an ADE
Patient signs, symptoms, and laboratory
values that followed the actual or missed
administration of a drug or blood product
Missed drug administration
Missed drug monitoring
Blood product administered
Sign or symptom
Abnormal laboratory value
Abnormal drug level
Abnormal test result (radiology,
Add new allergy
Change medication, dose, route
Escalate care (transfer to ICU, obtain
Start fluid restriction
Start new medication, biological product,
Start new procedure
Improvement from prior observation
Return to baseline
Physician-recognized drug reaction
Abnormal laboratory value
Abnormal sign or symptom
InterventionActions taken to treat the manifestations
of an ADE
Aggressive wound care
Re-open surgical wound intentionally
Collect specimen for culture
Place drain in surgical wound
Return to operating room
Response to treatmentInformation about patient status or
a specific manifestation documented in
a physician note following treatment of
Assessments Diagnosis of HAI
ADE, adverse drug event; HAI, hospital-associated infection.
J Am Med Inform Assoc 2011;18:491e497. doi:10.1136/amiajnl-2011-000187 493
Research and applications
a response to the treatment of the event (32% and 17%, respec-
tively). Only 58% of HAIs and 34% of ADEs had a physician
assessment, such as a diagnosis of an infection or a recognized
drug reaction, recorded in an electronic physician narrative.
Source of information
Information about each event attribute and the type of elec-
tronic physician narrative in which it was found is summarized
in table 4. To detect differences between the different subtypes
of infections, we grouped the HAI results according to SSIs,
LRTIs, and UTIs. Most types of narratives contained informa-
tion about at least one SSI. For all types of event attributes,
information about SSIs was found most often in discharge
summaries. Information was found most often in a discharge
summary for interventions to treat the SSI (92%). Emergency
department reports and admission history and physical reports
contained information about SSIs that were attributed to
previous hospitalizations. Information about SSIs that required
surgical intervention or draining of an abscess was found in
general surgery reports and procedural radiology reports,
respectively. Information about SSI-related physician assessment
(eg, diagnosis of infection) was found in discharge summaries
(42%), emergency department reports (17%), admit history/
physical reports (8%), and general consult notes (4%). The SSIs
were the only type of HAI with information contained in an
emergency department report, an admission history and phys-
ical report, a general surgery report, or a report about a radi-
ology-guided procedure. For manifestations of LRTIs, diagnostic
radiology reports contained information about more events than
discharge summaries (100% and 67%). Interventions to treat the
LRTI (67%), responses to this treatment (33%), and assessments
by a physician (50%) were found most often in discharge
summaries. General consult reports contained information
about patients who were transferred to a critical care unit
following cardiopulmonary arrest which had led to aspiration
pneumonia (a specific type of LRTI). Information about one
patient with an LRTI who died was found in a death summary
report. All attributes about the one UTI missed by CSS were
found in the patient’s discharge summary.
Like HAIs, discharge summaries contained information about
more ADEs than any other type of electronic physician narra-
tive. The most frequent type of attribute found was manifes-
tation of an ADE (64%), followed by an intervention to treat an
ADE (35%). Emergency department reports and admit history/
physical reports contained information about events that
manifested prior to admission. Information about several ADEs
(11%) had information found in general consult reports, such as
patients with analgesic-related oversedation or a reaction to
anesthesia. Diagnostic radiology reports had the least amount of
information about ADEs. Physician recognition of an ADE was
found most often in discharge summaries (27%) and in general
consult reports (9%).
Only 58% of HAIs and 34% of ADEs missed by CSS were
explicitly acknowledged in at least one physician narrative. The
information about the remaining AEs that were not explicitly
mentioned in physician narratives was spread out across
different phrases and/or documents.
Healthcare organizations use different methods to detect and
abstraction, and concurrent clinical surveillance. Prior studies
attributed differences between surveillance methods to the data
sources used by each method, to differences in the subject
matter expertise among human reviewers, and to cognitive
challenges faced by the reviewers.15 17 31 32Other differences,
such as timing, scope, and workflow of surveillance, may also
contribute to these differences.33We expected that MCR would
detect AEs missed by CSS and, taking advantage of these
differences, we could improve CSS identification of AEs. Because
agreement between MCR and CSS was less for ADEs than for
HAIs, the potential for improving CSS is greater for ADEs than
for HAIs. Integrating information from physician narratives
with CSS would potentially capture a greater proportion of
additional ADEs than HAIs.
Improving CSS with information from physician narratives
Bates et al suggest that integrating information from physician
narratives with automated surveillance methods would increase
the number of AEs detected.34Based on our findings, adding data
from physician narratives would have helped CSS detect
some, but not all, missed cases. Review of the phrases we
collected suggested that detection of LRTIs, SSIs, and ADEs
would improve if patient signs, symptoms, interventions, and
surveillance system, by manual chart review, or by both
Number of HAIs and ADEs in the study sample that were detected by either computerized
Type of event
Number (%) of events
detected by each method
Number (%) events detected by only one method or
CSSMCR CSS onlyBoth methods MCR only
ADE, adverse drug event; BSI, bloodstream infection; CSS, computerized surveillance system; HAI, hospital-associated infection; LRTI,
lower respiratory tract infection; MCR, manual chart review; SSI, surgical site infection; UTI urinary tract infection.
least one attribute found in an electronic physician narrative grouped by
category of attribute
Number of HAIs and ADEs found only by MCR that had at
Response to treatment
Information about a causative drug was included only when it was associated with another
ADE attribute in the same phrase. We did not include instances in which the causative drug
was mentioned outside of the context of an ADE manifestation, intervention, or response to
treatment. Causative drugs were not applicable (NA) to HAI surveillance for the purposes of
ADE, adverse drug event; HAI, hospital-associated infection; MCR, manual chart review.
494J Am Med Inform Assoc 2011;18:491e497. doi:10.1136/amiajnl-2011-000187
Research and applications
physician assessments from physician narratives were integrated
Using microbiology culture results and the urinary catheter
surveillance used at LDS Hospital, CSS detected all of the BSIs
and all but one of the UTIs in the study. Thus, detection of these
types of events would not benefit much from integration of data
from physician narratives. We did not find microbiology culture
results for the single UTI, the LRTIs, or several SSIs in either the
laboratory system or the physician narratives. The CSS missed
some deep incisional and organ space SSIs, because the specimen
was entered into the laboratory information system as
unstructured freetext as opposed to the expected coded format.
In the absence of microbiology data, the signs, symptoms,
radiographic evidence (for LRTIs and organ space SSIs), treat-
ment, and diagnoses contained in physician narratives could
serve as triggers for HAIs.
The CSS missed ADEs for the following reasons: (a) infor-
mation needed to trigger an alert was not available to the
system, (b) information was available to the system but no alert
was triggered, and (c) an assessment of a suspected ADE was not
documented by the clinical pharmacist in CSS. We encountered
two instances where an intervention (eg, administration of
vitamin K or naloxone) was mentioned in the physician narra-
tive but not recorded in the pharmacy system as either an order
or an administration event. Thus, physician narratives proved to
be an alternate source of information for medication-related
events that were not recorded electronically in the pharmacy
information system. However, addition of data from physician
narratives would not improve CSS for cases for which no alert
was generated or where the alert was generated but not
reviewed by the clinical pharmacist. For cases where no alert was
generated, time-driving the ADE logic and scanning the data
from all patients may be more effective rather than depending
on using data-driven triggers to activate the logic.
Content of physician narratives
Our analysis of physician narratives revealed challenges in using
narrative text to support CSS. Physicians may respond to AEs as
part of routine course of care and not document observations
and interventions with surveillance in mind.35In this study,
only 58% of HAIs and 34% of ADEs missed by CSS were
explicitly documented in dictated reports. If natural language
processing could detect these phrases, then CSS would most
likely be able to pick up these additional AEs. The lack of explicit
physician acknowledgment for the remaining AEs presents
a challenge for automated surveillance methods.17 35In many
AEs, supporting evidence for an LRTI, SSI, or ADE was distrib-
uted across multiple physician narratives. In the absence of
explicit recognition of the AE, CSS would need to handle
information from multiple places in the same document or from
multiple documents to identify ADEs and HAIs.
Sources of AE information
By examining the content of physician narratives, we identified
those that were likely to contain information about each type of
AE. Discharge summaries contained information about more
HAIs and ADEs than any other electronic physician narrative.
Discharge summaries would be a valuable source of information
for retrospective measurement and for confirmation of AEs
detected earlier in the admission by other methods. But their
benefit to prospective surveillance would be limited, since
discharge summaries are not available prior to discharge.
Number of HAIs and ADEs found only by MCR with information in each type of electronic physician narrative grouped by attribute category
Number (%) of events*
MCR data sources
Response to treatment
Response to treatment
Response to treatment
Causative drug (n¼34)
Response to treatment
*Percent of the number of SSIs, LRTIs, UTIs, and ADEs, respectively.
One ADE had drug, manifestation, and intervention in a single endoscopy procedure report.
ADE, adverse drug event; ED, emergency department; HAI, hospital-associated infection; LRTI, lower respiratory tract infection; MCR, manual chart review; SSI, surgical site infection; UTI
urinary tract infection.
4 (17)2 (8)eee 1 (4)10 (42)e
eeeee 1 (17)3 (50) 1 (17)
1 (1)1 (1)e1 (1)e8 (9)25 (27)1 (1)
J Am Med Inform Assoc 2011;18:491e497. doi:10.1136/amiajnl-2011-000187495
Research and applications
We had to look at the particular subtypes of HAIs to find
specific opportunities to improve HAI surveillance by CSS. In
order to improve surveillance of SSIs that required readmission,
CSS would need access to the information found in emergency
department reports and admission history and physical reports.
These reports contained information about signs, symptoms,
significant white blood cell counts, antimicrobial treatment,
impressions. To improve the surveillance of SSIs that occurred
within the current admission, CSS would need access to
information found in general surgery reports; these reports
contained phrases that suggested the presence of a ‘post-opera-
tive wound infection.’ Improving the detection of LRTIs would
require access to general consult reports, which contained signs,
symptoms, antimicrobial treatment, and physician impressions.
In addition to signs of pneumonia (important for LRTIs),
diagnostic radiology reports contained important evidence of
intra-abdominal and retroperitoneal abscesses, which were
important indicators of SSIs that required radiologically-guided
drainage. Information about one post-procedural LRTI was
found in a death summary report.
Information about outpatient ADEs was found in emergency
department reports and admission history and physical reports.
Information about anticoagulation-related bleeding events was
found in general surgery reports, radiology reports, and discharge
summaries. For example, one patient with repeated bleeding
episodes on Coumadin received an inferior vena cava filter,
which was documented in a radiology report because it was
placed under radiographic guidance. Another anticoagulation-
related gastrointestinal bleeding event was recorded in an
endoscopy report. The ADEs severe enough to require transfer to
the intensive care unit were mentioned in a general consult
report, which included general anesthesia-related events, cardiac
arrests secondary to cardiovascular medications, and opiate-
related sedation. Almost all ADEs involving narcotic analgesics
were mentioned in a general consult report, which contained
signs (eg, mental status changes and decreased respiratory rate),
response to naloxone, and physician assessments.
If the clinician did not document their assessment of a suspected
case in the CSS, we were unable to distinguish between false
positive cases and suspected AEs that were not reviewed. This is
an important area for additional investigation, since it would
affect the benefit obtained by the integration of additional data
from physician narratives.
Recommendations for future work
Physician narratives must be available in electronic form, so that
CSS can access their content. The ideal narrative for a concur-
rent system like CSS is the progress note, since it is typically
createddaily throughout a hospitalization.
implement electronic progress notes, we need to understand
what information about AEs is more likely to be recorded in
progress notes than other physician narratives.
Additional investigation of ADEs missed by CSS is needed to
troubleshoot the system and the surveillance workflow. In these
cases, improvements may be attained in the cognitive burden,
staffing, and prioritization of patient safety activities.
As public reporting requirements increase, providers must
consider the role of surveillance technologies. In this study, we
identified and described differences between two such systems,
including how each system used information from different
sources. Computerized surveillance system detection of LRTIs,
SSIs, and ADEs would improve if patient signs, symptoms,
interventions, and physician assessments from physician narra-
tives were integrated using technologies such as natural
Acknowledgments We would like to thank Vikrant Deshmukh, MSc, MS for his
expert advice in the design of the database application used for this study.
Funding This project was funded in part by an institutional medical informatics
training grant from the National Library of Medicine (contract number 5T
Competing interests None.
Ethics approval This study was approved by Intermountain Healthcare and the
University of Utah.
Provenance and peer review Not commissioned; externally peer reviewed.
Institute of Medicine. Performance Measurement: Accelerating Improvement.
Washington, DC: National Academies Press, 2006:1e16.
Pennsylvania Health Care Cost Containment Council. Hospital-Acquired
Infections in Pennsylvania. 2007. http://www.phc4.org/reports/hai/ (accessed 26
Lindenauer PK, Remus D, Roman S, et al. Public reporting and pay for performance
in hospital quality improvement. N Engl J Med 2007;356:486e96.
Fung CH, Lim YW, Mattke S, et al. Systematic review: the evidence that publishing
patient care performance data improves quality of care. Ann Intern Med
The Joint Commission. The Joint Commission Health Care Quality Data Download
Website. 2010. http://www.healthcarequalitydata.org/ (accessed 25 Nov 2010).
Lagu T, Lindenauer PK. Putting the public back in public reporting of health care
quality. JAMA 2010;304:1711e12.
Klompas M, Yokoe DS. Automated surveillance of health care-associated infections.
Clin Infect Dis 2009;48:1268e75.
Evans RS, Larsen RA, Burke JP. Computer surveillance of hospital-acquired
infections and antibiotic use. JAMA 1986;256:1007e11.
Classen DC, Pestotnik SL, Evans RS, et al. Computerized surveillance of adverse
drug events in hospital patients. JAMA 1991;266:2847e51.
Bates DW, Pappius E, Kuperman GJ, et al. Using information systems to measure
and improve quality. Int J Med Inform 1999;53:115e24.
Frank L, Galanos H, Penn S, et al. Using BPI and emerging technology to improve
patient safety. J Healthc Inf Manag 2004;18:65e71.
Chen ES, Wajngurt D, Qureshi K, et al. Automated real-time detection and
notification of positive infection cases. AMIA Annu Symp Proc 2006:883.
Kilbridge PM, Campbell UC, Cozart HB, et al. Automated surveillance for adverse
drug events at a community hospital and an academic medical center. J Am Med
Inform Assoc 2006;13:372e7.
Ferranti JM, Langman MK, Tanaka D, et al. Bridging the gap: leveraging business
intelligence tools in support of patient safety and financial effectiveness. J Am Med
Inform Assoc 2010;17:136e43.
Jha AK, Kuperman GJ, Teich JM, et al. Identifying adverse drug events:
development of a computer-based monitor and comparison with chart review and
stimulated voluntary report. J Am Med Inform Assoc 1998;5:305e14.
Fiszman M, Chapman WW, Aronsky D, et al. Automatic detection of acute
bacterial pneumonia from chest x-ray reports. J Am Med Inform Assoc
Murff HJ, Forster AJ, Peterson JF, et al. Electronically screening discharge
summaries for adverse medical events. J Am Med Inform Assoc 2003;10:339e50.
Melton GB, Hripcsak G. Automated detection of adverse events using natural
language processing of discharge summaries. J Am Med Inform Assoc
Gardner RM, Pryor TA, Warner HR. The HELP hospital information system: update
1998. Int J Med Inf 1999;54:169e82.
20.Special issue: The SENIC Project. Am J Epidemiol 1980;111:465e653.
Evans RS, Gardner RM, Bush AR. Development of a computerized infectious disease
monitor (CIDM). Comput Biomed Res 1985;18:103e13.
Evans RS, Pestotnik SL, Classen DC, et al. Development of a computerized adverse
drug event monitor. Proc Annu Symp Comput Appl Med Care 1991:23e7.
Jacobson JA, Burke JP, Kasworm E. Effect of bacteriologic monitoring of urinary
catheters on recognition and treatment of hospital-acquired urinary tract infections.
Infect Control 1981;2:227e32.
Classen DC, Pestotnik SL, Evans RS, et al. Adverse drug events in hospitalized
patients: excess length of stay, extra costs, and attributable mortality. J Am Med
496J Am Med Inform Assoc 2011;18:491e497. doi:10.1136/amiajnl-2011-000187
Research and applications
25. Download full-text
Weissman JS, Rothschild JM, Bendavid E, et al. Hospital workload and adverse
events. Med Care 2007;45:448e55.
Iezzoni LI, Daley J, Heeren T, et al. Identifying complications of care using
administrative data. Med Care 1994;32:700e15.
Bates DW, O’Neil AC, Petersen LA, et al. Evaluation of screening criteria for adverse
events in medical patients. Med Care 1995;33:452e62.
Garner JS, Jarvis WR, Emori TG, et al. CDC definitions for nosocomial infections,
1988. Am J Infect Control 1988;16:128e40.
Gerberding J, Gaynes R, Horan T, et al. National Nosocomial Infections Surveillance
(NNIS) system report, data summary from January 1990-May 1999, issued June
1999. Am J Infect Control 1999;27:520e32.
Kohn LT, ed. To Err is Human: Building a Safer Health System. Washington, DC:
National Academy Press, 1999:33.
Kongkaew C, Noyce PR, Ashcroft DM. Hospital admissions associated with adverse
drug reactions: a systematic review of prospective observational studies. Ann
Phansalkar S, Hoffman JM, Hurdle JF, et al. Understanding pharmacist decision
making for adverse drug event (ADE) detection. J Eval Clin Pract 2009;15:266e75.
Michel P, Quenon JL, de Sarasqueta AM. Comparison of three methods for
estimating rates of adverse events and rates of preventable adverse events in acute
care hospitals. BMJ 2004;328:199.
Bates DW, Evans RS, Murff H, et al. Detecting adverse events using information
technology. J Am Med Inform Assoc 2003;10:115e28.
Nebeker JR, Barach P, Samore MH. Clarifying adverse drug events: a clinician’s
guide to terminology, documentation, and reporting. Ann Intern Med
Every month look out for intriguing cases on a different speciality:
To submit your cases and read others in your specialty become a Fellow. casereports.bmj.com
J Am Med Inform Assoc 2011;18:491e497. doi:10.1136/amiajnl-2011-000187497
Research and applications