Content uploaded by Peter Nielsen
Author content
All content in this area was uploaded by Peter Nielsen on Jul 23, 2018
Content may be subject to copyright.
www.jcrinc.com
Improvement from
Front Oce to Front Line
August 2015
Volume 41 Number 8
Computerized Provider
Order Entry as a
Cost-Savings Strategy
Features
Information Technology
e Value of Computerized Provider Order Entry: Is It Time for
the Debate to Be Over?
Implementing Computerized Provider Order Entry in Acute Care
Hospitals in the United States Could Generate Substantial Savings
to Society
Medication Safety
Advancing Medication Safety: Establishing a National Action Plan
for Adverse Drug Event Prevention
Implementation Science
Facilitation of a Multihospital Community of Practice to
Increase Enrollment in the Hospital to Home National Quality
Improvement Initiative
Performance Measures
Accuracy of the Adverse Outcome Index: An Obstetrical
Quality Measure
Departments
Forum
Connecting Patients and Clinicians: e Anticipated Eects
of Open Notes on Patient Safety and Quality of Care
“The probability that
implementing CPOE in acute
care hospitals would, on
average, and using typical
estimates of implementation
costs, improve health and
yield cost savings to society
exceeds 99%.”
—Implementing Computerized Provider
Order Entry in Acute Care Hospitals in
the United States Could Generate
Substantial Savings to Society
(p. 341)
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015
370
Pregnancy and childbirth are the leading causes of hospital
admissions for women in the United States. In 2009 “live
born infant” was the most commonly coded diagnosis, account-
ing for more than 10% of all hospital stays.1 In 2012 the Insti-
tute for Healthcare Improvement (Cambridge, Massachusetts)
reported that obstetrical adverse events of some sort occur in
9% of all deliveries.2 Deliveries with complications are costly;
in 2008 they accounted for $17.4 billion, which represents 5%
of total hospital costs in the United States for that year. A hospi-
tal stay for a pregnancy with complications is about 50% more
costly than one without complications.3
In obstetrics, a nationally accepted set of quality indicators
for patient safety was not available until a multidisciplinary
panel of experts met in June 2001 in an attempt to develop
an acceptable set of measures.4 Professional participants in the
subsequent consensus conferences developed a set of 10 adverse
outcome measures, called the Adverse Outcome Index (AOI).
e US Department of Defense and the Harvard Risk Man-
agement Strategies Foundation (Cambridge) rst examined the
impact of team training and teamwork on obstetrical care and
the AOI in a multicenter labor and delivery study involving 15
institutions and more than 28,000 patients, in 2002.5 Subse-
quently, further evaluation of the AOI as a proposed inpatient
quality measure was advocated.6 e National Perinatal Infor-
mation Center (NPIC; Providence, Rhode Island) developed
hospital discharge data–based algorithms combined with a small
set of supplemental patient data for calculation of the AOI. e
NPIC administrative algorithm is an index of 10 adverse out-
comes, obtained retrospectively from coded discharge data. is
generates a list of births with one or more adverse outcomes,
with a designated list of International Classication of Diseases,
Ninth Revision, Clinical Modication (ICD-9-CM)/Diagnosis-
Related Group (DRG) as inclusion and exclusion criteria.
NPIC uses the AOI to track adverse obstetrical outcomes na-
tionwide.7 To date, only one study evaluated the AOI as derived
from the NPIC algorithms. e nding of variable accuracy for
hospital perinatal adverse event measurement in this small-N
Performance Measures
Accuracy of the Adverse Outcome Index: An Obstetrical
Quality Measure
Lisa M. Foglia, MD; Peter E. Nielsen, MD; Eileen A. Hemann, EdD; Suzan Walker, MPH; Jason A. Pates, MD;
Peter G. Napolitano, MD; Shad Deering, MD
Article-at-a-Glance
Background: In obstetrics, a nationally accepted set of
quality indicators for patient safety was not available in the
United States until the development of a set of 10 adverse
outcome measures—the Adverse Outcome Index (AOI).
e National Perinatal Information Center (NPIC) devel-
oped hospital discharge data–based algorithms combined
with a small set of supplemental patient data for calculation
of the AOI. A study was conducted to determine the speci-
city, sensitivity, positive predictive value (PPV), and nega-
tive predictive value (NPV) of the AOI by using the National
Perinatal Information Center (NPIC) algorithm.
Methods: A retrospective chart review of 4,252 obstetri-
cal and neonatal charts from 2003 through 2007 was per-
formed. NPIC denitions were compared with the “gold
standard”—chart review.
Results: A total of 229 deliveries among the 4,000 random-
ly selected charts had at least one adverse outcome, reecting
an AOI of 5.7%. For detection of the 10 adverse outcomes
within the AOI, the overall sensitivity of the AOI was 81.7%,
specicity was 98.2%, PPV was 86.3%, and NPV was 97.4%.
e Kappa value for agreement between the coded charts and
the chart review was 0.82 (standard deviation = 0.01, 95%
condence interval [CI] = 0.80–0.85), which is considered
very good.
Discussion: e AOI is highly reliant on accurate coding
and provider documentation and requires validation with
manual chart review. Concurrent chart review improves the
accuracy of the AOI. Caution is advised when using the AOI
as an exclusive measure of assessing obstetric quality because
it may be heavily inuenced by a single outcome measure;
perineal laceration rates represented twice the frequency of
all other outcomes combined. e AOI should be modied
to better measure preventable adverse events and include a
means of accounting for preexisting conditions.
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015 371
(33 mother/infant pairs) study led the authors to advocate for
better understanding and evaluation of the individual metrics.8
e purpose of the current study was to determine the specici-
ty, sensitivity, positive predictive value (PPV), and negative pre-
dictive value (NPV) of the AOI as it is currently calculated on
a larger sample and determine its accuracy for our institution.
Methods
Setting
is study was performed in 2011 at a tertiary care military
health care facility in the Pacic Northwest, with an average
delivery rate of 200 births per month. is was a retro spective
chart review of selected delivery and associated birth records
from 2003 through 2007. e charts were reviewed for adverse
outcomes using the NPIC AOI denitions, with their respec-
tive inclusion and exclusion criteria.
Manual Chart review
e AOI assesses a dichotomous state (such as the presence
or absence of an adverse event), when the true state is known
by some “gold standard.” In this study, the gold standard was
a manual chart review of the patient’s electronic health record
(EHR) by one of four board-certied obstetrician/gynecolo-
gists [L.M.F., J.A.P., P.G.N., S.D.]. Chart review ndings were
compared to the adverse outcomes identied using the NPIC
administrative algorithm. We determined the sensitivity, spec-
icity, PPV, and NPV of the AOI for each adverse outcome.
SaMple and proCeSS for Chart review
On the basis of a sensitivity of the computer-determined AOI
relative to chart-reviewed AOI of 90%, with a 95% condence
interval [CI] precision +/– 5%, 4,000 charts were needed for
this review. Charts were randomly selected from all births in the
study period, using the Excel™ randomization function (rand-
between, 1,7539; Microsoft™, Redmond, California), which
assigns a unique number to each hospital birth. e list was
sorted in ascending order, and the rst 2,000 births were select-
ed for chart review for each of the two time periods. We also in-
cluded 252 charts from this time period that were identied by
the NPIC algorithm as having an adverse event. is resulted in
a total of 4,252 charts for review (Figure 1, above) We elected to
include the additional 252 charts with at least one adverse out-
come identied by the NPIC algorithm to evaluate the accuracy
of all records with adverse outcomes identied by NPIC. Charts
(History and Physical, Delivery Summary, and Discharge Sum-
mary) were reviewed to verify correct identication of one or
Figure 1. All hospital births from two designated time periods were randomized to yield 2,000 mother/infant charts for a total of 4,000 charts. After this list
was obtained, it was compared with the Adverse Outcome Index (AOI) obtained from the National Perinatal Information Center (NPIC) for the same time
period. Any births from the NPIC list (AOI present, when an administrative algorithm was used) that were not selected in the randomized samples were added
to the chart review, yielding 4,252 mother/infant records to be reviewed for adverse outcomes.
Selection of Study Population
7,539 Hospital
deliveries during
designated study time
periods
3,530 deliveries between
10/01/01–09/30/03
randomized to yield
2,000 births charts to be
reviewed for AOIs
4,009 deliveries between
01/01/05–12/31/06
randomized to yield
2,000 births charts to be
reviewed for AOIs
7,539 Hospital
deliveries during
designated study time
periods
3,530 deliveries between
10/01/01 and 09/30/03
randomized to yield
2,000 birth charts to be
reviewed for AOIs
4,009 deliveries between
01/01/05 and 12/31/06
randomized to yield
2,000 birth charts to be
reviewed for AOIs
503 NPIC deliveries
with AOIs using
the administrative
algorithm
252 deliveries not
included in randomized
birth lists when
compared with NPIC
AOI cases
503 NPIC deliveries
with AOIs using
the administrative
algorithm
252 deliveries not
included in randomized
birth lists when
compared with NPIC
AOI cases
AOI found were
validated by
Obstetrical Physician
4,252 total cases
reviewed for AOIs
AOIs found were
validated by
Obstetrical Physician
4,252 total cases
reviewed for AOIs
++=
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015
372
more of the 10 adverse outcomes in the AOI compared with
the NPIC algorithm–identied cases. Additional notes were re-
viewed, as necessary to verify or dispute the information iden-
tied by NPIC (for example, pre- and posttransfusion notes for
patients receiving blood products). e data collection sheet is
provided in Appendix 1 (available in online article). e PPV
and NPV were calculated for the NPIC administrative algo-
rithm–identied adverse outcomes. is study was approved by
the hospital Institutional Review Board, and a waiver of consent
was obtained.
Results
Maternal delivery and neonatal reCordS
We reviewed 4,252 delivery and neonatal records. e overall
sensitivity of the AOI was 81.7%, PPV was 86.3%, specicity
was 98.2%, and NPV was 97.4%. Patients with one or more
adverse outcome were more likely to have larger birth weights,
higher rates of operative vaginal delivery, and lower rates of
cesarean delivery (Table 1, above).
adverSe outCoMeS
A total of 530 adverse outcomes occurred among 481 de-
liveries in the 4,252 records reviewed. A total of 229 deliveries
among the 4,000 randomly selected charts had at least one ad-
verse outcome, reecting an AOI of 5.7% within this random
sample of records. Four hundred forty-one deliveries had one
adverse event, 33 deliveries had two adverse events, 6 deliveries
had three adverse events, and 1 delivery had ve adverse events.
e most frequent adverse event was 3rd or 4th degree lacera-
tion, followed by maternal blood transfusion, birth trauma, in-
fant admission to the neonatal ICU (NICU), maternal admit to
ICU, ve-minute Apgar of < 7, maternal return to the operating
room, uterine rupture, and term neonatal death. ere were no
maternal deaths per the chart review or NPIC algorithm during
the study period (Figure 2, page 374; color version available in
online article).
e Kappa value for agreement between the coded charts
and the chart reviews, as calculated using GraphPad (http://
www.graphpad.com/quickcalcs/kappa1.cmf), was 0.82 (stan-
dard deviation = 0.01, 95% CI = 0.80–0.85). Table 2 (page
373) lists the strength of agreement of the indicators. Measures
of accuracy were calculated for 9 of the 10 adverse outcomes
(maternal death excluded). Both term neonatal deaths were cor-
rectly identied using the algorithm and conrmed by chart
review. e algorithm had a sensitivity of 80% for three out-
comes: 3rd or 4th degree laceration, uterine rupture, and birth
trauma. e algorithm had the lowest sensitivity (≤ 50%) for
maternal return to the operating room, maternal blood trans-
fusions, maternal admission to the ICU, infant admission to
NICU, and a ve-minute Apgar score of < 7. e specicity
of the algorithm was greater than 99% for all adverse outcome
indicators (Table 2). e NPV was greater than 90% for all
indicators. However, the PPV of the algorithm for infant ad-
missions to the NICU, ve-minute Apgar scores of < 7, ma-
ternal return to the operating room, and uterine rupture was
less than 70%. e adverse outcome indicators with the highest
PPV were term neonatal death, maternal admissions to ICU,
maternal blood transfusion, 3rd or 4th degree laceration, and
birth trauma (Table 2).
Discussion
e nding of this study supports the wide variability in accu-
racy of the PPVs for the AOI among the 10 adverse outcomes,
as found by Walker, Strandjord, and Bennedetti.8 eir study
identied a PPV of 0 for term neonatal death due to incor-
rect birth weight codes, as well as relatively high PPVs for ve-
Table 1. Characteristics of the Sampled Population
Population with One or More
Adverse Events (N = 481)
Population Without an Adverse
Event (N = 3,771) P Value
Gestation Age-weeks (mean + SD) 38.7 + 2.7 38.6 + 2.6 .476
Birth Weight-grams (mean + SD) 3,431 + 730 3,331 + 641 .003
Method of Delivery Frequency Percentage Frequency Percentage
Spontaneous Vaginal 222 46.2 2,632 69.8 < .001
Forceps 145 30.1 101 2.7 < .001
Vacuum 47 9.8 85 2.3 < .001
Cesarean 67 13.9 953 25.3 < .001
Total 481 100.0 3,771 100.0
SD, standard deviation.
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015 373
minute Apgar scores and NICU admission—which were due
to improved methods to document Apgar scores and chang-
es in NICU billing codes. In our study, the most accurately
identied indicators were term neonatal death, uterine rup-
ture, birth trauma, and 3rd or 4th degree laceration. e least
accurately identied events were term infants admitted to the
NICU, ve-minute Apgar scores of < 7, and maternal return to
the operating room. Because there were no maternal deaths in
our population, the accuracy of this indictor could not be de-
termined. Maternal admission to the ICU, NICU admissions,
and identication of uterine rupture, which heavily relied on
provider documentation and the resultant codes used for diag-
noses and billing, were the most dicult to accurately identify
via the algorithm.
Examining the list of codes in the NPIC algorithm for com-
prehensiveness is warranted. For example, birth trauma codes
included in the NPIC denition are subdural and cerebral
hemorrhage (767.00), epicranial subaponeurotic hemorrhage
(767.11), other injuries to the skeleton (767.30), injury to spine
and spinal column (767.40), facial nerve injury (767.50), inju-
ry to brachial plexus (767.60), and other cranial and periph-
eral nerve injury (767.70). Not included in the algorithm but
identied in our chart review were clavicular fracture (767.20)
and right hip dislocation from breech delivery (754.32), both
of which were documented as birth trauma. Adding these diag-
noses to the list captured by the AOI for this adverse outcome
should be considered.
e ability to distinguish between a maternal return to
the operating room to address a complication versus an inde-
pendent second procedure performed immediately following
delivery (for example, postpartum bilateral tubal ligation) was
challenging for both the NPIC algorithm and chart review.
Medical record reviews proved time consuming for this measure
because the documentation was found in text elds, as opposed
to data elements, which could be queried.
System-specic issues can also aect the accuracy of the AOI,
which further indicated the importance of concomitant medical
record reviews to identify areas aecting the AOI accuracy. For
example, neonatal admissions to the NICU are subject to hospi-
tal-specic admission policies and criteria. At our institution, it
was dicult to identify admission to the NICU versus interme-
diate care nursery because they are physically located in the same
place and are both indicated as “NICU” in the EHR, which can
overinate the NICU admissions identied by the algorithm.
For example, an infant admitted for observation because of ma-
ternal chorioamnionitis appeared as a NICU admission in the
NPIC algorithm. Although these infants required a higher level
of care than routine postpartum care, they were not of the acuity
level required for a NICU admission and were documented as
such on manual chart review. Revising this discrete-level indica-
tor to a severity ordinal scale with a cuto threshold to include
only the most severe admission diagnoses to the NICU should
be considered to rene the accuracy of this outcome.
In 2011 the National Quality Forum endorsed a new quality
measure, Healthy Term Newborn, stewarded by the California
Maternal Quality Care Collaborative.9 is measure, a compos-
ite of unexpected neonatal complications that developed during
birth or neonatal care, is included in the 29–clinical quality
Table 2. Characteristics of the National Perinatal Information Center Adverse Outcome Index Algorithm
Compared to Chart Audit (N = 4,252)
Adverse Outcome Sensitivity (%) Specicity (%) PPV (%) NPV (%) Kappa Coefcient
Kappa
Strength of
Agreement* 95% CI SE
Maternal Death*
Neonatal Death 100.0 100.0 100.0 100.0 1.00 Perfect 1.00–1.00 0.00
3rd or 4th Degree
Laceration 98.0 99.6 95.2 99.8 0.96 Very Good 0.95–0.98 0.01
Birth Trauma 83.7 99.8 80.4 99.8 0.82 Very Good 0.74–0.90 0.04
Uterine Rupture 100.0 99.9 66.7 100.0 0.80 Very Good 0.53–1.07 0.14
Maternal Admission to ICU 47.4 100.0 100.0 99.8 0.64 Good 0.44–0.85 0.11
Apgar < 7 at 5 minutes 26.3 99.8 38.5 99.7 0.31 Fair 0.11–0.52 0.11
Infant Admission to NICU 30.0 99.4 25.0 99.5 0.27 Fair 0.13–0.41 0.07
PPV, positive predictive value; NPV, negative predictive value; CI, condence interval; SE, standard error.
* Poor = < 2.0; fair = .20–<.40; moderate = .40–< .60; good = .60–< .80; very good = .80–1.00.
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015
374
Figure 2. Charts (history and physical, delivery summary, and discharge summary; “Gold Standard/Electronic Health Record [EHR] Chart Review”)
were reviewed to verify correct identication of 1 or more of the 10 adverse outcomes in the Adverse Outcome Index compared with the National Perinatal
Information Center (NPIC) algorithm–identied cases. Positive predictive values (PPVs), negative predictive value (NPVs) and subsequent specicity and
sensitivity values for detecting the 10 adverse outcomes are shown. e overall sensitivity of the AOI was 81.7%, specicity was 98.2%, PPV was 86.3%, and
NPV was 97.4%. TP, true positive; FP, false positive; TN, true negative; FN, false negative; OR/L&D, operating room/labor and delivery.
Specicity and Sensitivity Values for 4,252 Delivery and Neonatal Records
Positive Negative
Positive 433 69
PPV
86.25%
TP/(TP + FP)
percentage
Negative 97 3,653
NPV
97.41%
TN/(FN + TN)
percentage
Sensitivity Specificity
81.70%98.15%
TP/(TP + FN) TN/(FP + T N)
percentagepercentage
Positive Negative Positive Negative
Positive 0 0
PPV
N/A
Positive 4 4
PPV
50.00%
Negative 0 4,252
NPV
100.00%
Negative 4 4,240
NPV
99.91%
Sensitivity Specificity Sensitivity Specificity
0.00%100.00%50.00% 99.91%
Positive Negative Positive Negative
Positive 2 0
PPV
100.00%
Positive 9 27
PPV
25.00%
Negative 0 4,250
NPV
100.00%
Negative 21 4,195
NPV
99.50%
Sensitivity Specificity Sensitivity Specificity
0.00%100.00%30.00%99.36%
Positive Negative Positive Negative
Positive 4 2
PPV
66.67%
Positive 5 8
PPV
38.46%
Negative 0 4,246
NPV
100.00%
Negative 14 4,225
NPV
99.67%
Sensitivity Specificity Sensitivity Specificity
100.00%99.95%26.32%99.81%
Positive Negative Positive Negative
Positive 9 0
PPV
100.00%
Positive 25 1
PPV
96.15%
Negative 10 4,233
NPV
99.76%
Negative 33 4,193
NPV
99.22%
Sensitivity Specificity Sensitivity Specificity
47.37%100.00%43.10%99.98%
Positive Negative Positive Negative
Positive 41 10
PPV
80.39%
Positive 334 17
PPV
95.16%
Negative 8 4,193
NPV
99.81%
Negative 7 3,894
NPV
99.82%
Sensitivity Specificity Sensitivity Specificity
83.67%99.76%97.95%99.57%
NPIC Administrative
Algorithm
Infant Admit to NICU
Gold Standard/EHR
Chart Review
NPIC Administrative
Algorithm
3rd/4th Degree
Lacerations
Gold Standard/EHR
Chart Review
Apgars < 7 @ 5 minutes
Gold Standard/EHR
Chart Review
NPIC Administrative
Algorithm
Maternal Blood
Transfusions
Gold Standard/EHR
Chart Review
Maternal Return to
OR/L&D
Gold Standard/EHR
Chart Review
NPIC Administrative
Algorithm
NPIC Administrative
Algorithm
Neonatal Death
Gold Standard/EHR
Chart Review
NPIC Administrative
Algorithm
Uterine Rupture
Gold Standard/EHR
Chart Review
NPIC Administrative
Algorithm
Maternal Admit to ICU
Gold Standard/EHR
Chart Review
NPIC Administrative
Algorithm
Birth Trauma
Gold Standard/EHR
Chart Review
NPIC Administrative
Algorithm
NPIC Administrative
Algorithm
Overall
Gold Standard/EHR
Chart Review
NPIC Administrative
Algorithm
Maternal Death
Gold Standard/EHR
Chart Review
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015 375
measure set for Stage 2 of the Center for Medicare & Medicaid
Services EHR incentive program.10 Although not risk adjusted,
the measure does exclude preexisting conditions, such as multi-
ple gestations, preterm birth, congenital anomalies, growth re-
striction, and fetuses aected by selected maternal conditions.
In addition, the numerator straties the severity of complica-
tions. e severe category refers to death, a ve-minute Apgar
score of ≤ 3, and diagnosis codes for major birth injuries or
major neurologic, pulmonary, or infectious complications. e
moderate category refers to those with lesser diagnoses resulting
in longer lengths of stay but a low likelihood of medium, or
long-term morbidity, such as transient tachypnea of the new-
born. Feedback to the measure developers resulted in an update
to the measure to change the numerator to capture “Unex-
pected Newborn Complications” in those newborns without
preexisting complications who experience a severe or moderate
morbidity.11 is measure is complementary to the AOI, as it
provides a more precise neonatal outcome metric. Alternately,
this measure could replace components of the AOI (ve-minute
Apgar scores of < 7, birth trauma and infant admission to the
NICU) because it includes all these components, straties birth
injuries by severity, and excludes preexisting conditions.
Coding did not capture Apgar scores of < 7 at ve minutes
because codes do not exist for low Apgar scores; identication
required a manual chart review. With the national emphasis on
EHRs, collection of prospective data about obstetric outcomes
should be considered. Maternal blood transfusion was another
indicator aected by the maturity of our coding system. Before
June 2006, blood transfusions were not coded in discharge pa-
perwork at all, accounting for some of the loss of accuracy in
this indicator. After this date, there was a high correlation be-
tween the accuracy of the coding and the chart audit. Data for
sensitivity, specicity, PPV, and NPV before and after the cod-
ing change are shown in Figure 3 (above, right; color version
available in online article). e inclusion and exclusion criteria
for this indictor may also need to be reconsidered with a time
threshold from delivery. Blood transfusions associated with the
most serious adverse outcomes were identied in conjunction
with admission to the operating room or admission to ICU
within the rst 24 hours after delivery. Transfusions after the
rst 24 hours were primarily for symptomatic anemia.
Finally, the largest total number of adverse outcomes was
3rd or 4th degree laceration. Identication relied heavily on
the coder’s interpretation of physician documentation. Lacer-
ations were primarily mismatched when the physician docu-
mented a 2nd degree laceration with extension into the rectal
capsule (“partial 3rd degree laceration”) and the coding was for
a second-degree laceration instead of a 3rd degree laceration,
which encompasses a wide variation in the amount of sphinc-
ter damage (partial disruption of the external anal sphincter to
complete disruption of the internal and external anal sphinc-
ter). ese may be of varying clinical signicance with regard to
adverse outcomes.
Because perineal lacerations are associated exclusively with
vaginal delivery, interpretation of a high AOI may lead to erro-
neous conclusions that vaginal deliveries are less desirable than
cesarean deliveries. A strategy to lower the AOI by increasing
cesarean deliveries fails to account for the short- and long-term
risks associated with future pregnancies, such as uterine rupture
and placental implantation abnormalities, and their attendant
increased risk of maternal blood transfusion, hysterectomy, ICU
admission, and maternal death.12,13 erefore, the AOI should
be used in combination with other quality indicators that ac-
knowledge dierences in the types, rates, and severity of adverse
outcomes, depending on the mode of delivery. is will help
avoid inadvertently driving practice patterns toward increasing
Figure 3. Before June 2006, blood transfusions were not coded in discharge
paperwork at all, accounting for some of the loss of accuracy in this indicator.
After June 2006, there was a high correlation between the accuracy of
the coding and the chart review. Data for sensitivity, specicity, positive
predictive value (PPV), and negative predictive value (NPV) before and after
the coding change are shown for the National Perinatal Information Center
(NPIC) algorithm–identied cases. EHR, electronic health record; TP, true
positive; FP, false positive; TN, true negative; FN, false negative.
Maternal Blood Transfusion Sensitivity and
Specicity Before and After 2006
Positive Negative
Positive 1 1
PPV
50.00%
TP/(TP = FP)
percentage
Negative 17 3,552
NPV
99.60%
TN/(FN + TN)
percentage
Sensitivity Specificity
5.56%99.98%
TP/(TP + FN) TN/(FP + TN)
percentagepercentage
Maternal Blood
Transfusion before
June 2006
Gold Standard/EHR
Chart Review
NPIC
Administrative
Algorithm
Positive Negative
Positive 24 0
PPV
100.00%
TP/(TP = FP)
percentage
Negative 16 641
NPV
99.62%
TN/(FN + TN)
percentage
Sensitivity Specificity
60.00%100.00%
TP/(TP + FN) TN/(FP + TN)
percentagepercentage
Maternal Blood
Transfusion June
2006 and after
Gold Standard/EHR
Chart Review
NPIC
Administrative
Algorithm
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015
376
cesarean delivery rates to reduce rates of 3rd and 4th degree lac-
erations and subsequently reduce an institution’s AOI. Alterna-
tively, separate AOIs for vaginal and cesarean deliveries could
be considered because some complications are unique to each
mode of delivery. Finally, because the optimal operative delivery
rate (cesarean and operative vaginal) relative to adverse outcome
rate is not known, additional study is required to determine
what operative delivery rate optimizes clinical outcomes. It is
clear that operative delivery rates of both zero and 100% would
result in unacceptable rates of maternal and/or neonatal mor-
bidity and mortality.
e AOI is signicantly inuenced not only by high-volume
outcomes signicantly aecting the AOI but by events that may
represent appropriate medical care for preexisting and/or latent
or underlying patient conditions. For example, in an anemic pa-
tient undergoing a repeat cesarean delivery, a blood transfusion
may be medically appropriate and not indicative of an untow-
ard medical event. erefore, labeling all blood transfusions as
adverse events might have the unintended consequence of cast-
ing a negative connotation on a medically appropriate interven-
tion. e AOI also fails to account for adverse events that are
preventable by modiable patient lifestyle factors and preexist-
ing maternal and fetal medical/surgical comorbidities—such as
hypertension, maternal obesity, control of gestational/pregesta-
tional diabetes and its eect on birth weight, preexisting anemia,
maternal bleeding disorders, maternal collagen disorders, and
prior cesarean delivery and abnormal placentation—that sub-
stantially contribute to the risk of certain complications, One
of the major criticisms of the AOI is that many of the indicators
need to be risk adjusted for patient case mix.6 Discussions on the
inclusion and exclusion criteria of the index to make the AOI
more clinically relevant and useful are strongly recommended.
When any measurement instrument is used to collect data,
the users must be cognizant of the accuracy of the resulting
data, as assessed by validation. Some outcomes are easily ex-
tracted using coded data, while others require secondary data
sets or chart review by a provider for complicated cases. e
AOI can be used by health care organizations as one of many
tools in their quality arsenal. As with any tool, validation im-
proves the accuracy of the index and identies opportunities
for improvement. A focus on improved documentation in the
medical record through concurrent review of the chart prior to
patient discharge with direct feedback to physicians and nurses
will improve data accuracy. In addition, as adverse outcomes are
rare obstetrical events, several studies have developed indices of
perinatal outcomes, shifting the focus of measurement to nor-
malcy indicators instead of rare obstetrical outcomes.13,14
Summary
e AOI can be used as a quality benchmarking tool, but it has
some limitations. First, it is heavily inuenced by high-volume
outcomes, such as perineal lacerations. As with any outcome
assessment tool, it does not stratify birth injuries by severity or
exclude preexisting conditions that may increase the chances of
an adverse outcome. Also, the AOI is not adjusted to account
for the short- and long-term risks associated with cesarean de-
liveries. Preexisting risks (such as multiple previous cesarean
deliveries with subsequent abnormal placentation) may place
a patient at greater risk for an unpreventable adverse outcome
assessed by the AOI, in comparison to a healthy nullipara. Fi-
nally, it is highly reliant on the expertise of hospital coding and
pro vider documentation and requires validation with a repre-
sentative sam ple of chart audits.
We recommend that health care organizations using NPIC
algorithms undertake data validation using concurrent chart
reviews, as we have reported here, to guide improvement of pa-
tient care and safety. J
The work described in this article was supported by the TRICARE Management
Activity, Falls Church, Virginia. The views expressed herein are those of the authors
and do not reect the ofcial policy of the Department of the Army, the Department
of Defense, or the US Government. The National Perinatal Information Center is
the contract quality measurement vendor for all hospitals in the military healthcare
system.
Lisa M. Foglia, MD, is Residency Program Director, Department
of Obstetrics and Gynecology, Madigan Army Medical Center, Ta-
coma, Washington. Peter E. Nielsen, MD, formerly Chief of Clini-
cal Operations, Western Regional Medical Command, and Obstetrics
and Gynecology Consultant to US Army Surgeon General, Joint
Base Lewis-McChord, Washington, is Commander, General Leon-
ard Wood Army Community Hospital, Fort Leonard Wood, Missouri.
Eileen A. Hemann, EdD, is Perinatal Clinical Outcomes Research
Nurse, Department of Obstetrics and Gynecology, Madigan Army
Medical Center. Suzan Walker, MPH, is Perinatal Database Manager
and Regional Perinatal Network Coordinator, Pediatrics Division of
Neonatology, University of Washington School of Medicine, Seattle.
Jason A. Pates, MD, is Chief, Division of Obstetrics, Department of
Obstetrics and Gynecology, Madigan Army Medical Center. Peter G.
Napolitano, MD, is Program Director, Maternal-Fetal Medicine Fel-
lowship, and Director, Division of Maternal-Fetal Medicine, Depart-
ment of Obstetrics and Gynecology, Madigan Army Medical Center.
Shad Deering, MD, is Assistant Dean of Simulation Education and
Deputy Medical Director, Uniformed Services University of the Health
Sciences Simulation Center, and Chair, US Army Central Simulation
Committee, Uniformed Services of the Health Sciences, Bethesda,
Maryland. Please address correspondence to Peter E. Nielsen,
peter.e.nielsen.mil@mail.mil.
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015 377
References
1. Agency for Health Care Policy and Research. Most Common Diagnoses and
Procedures in U.S. Community Hospitals, 1996. Elixhauser A, Steiner CA.
AHCPR Publication No. 99-0046. Oct 1999. Accessed Jun 30, 2015. http://
www.hcup-us.ahrq.gov/reports/natstats/commdx/commdx.htm.
2. Gullo S. How-to-Guide: Prevent Obstetrical Adverse Events. Cambridge, MA:
Institute for Healthcare Improvement, 2011. Accessed Jun 30, 2015. http://
www.ihi.org/offerings/Documents/ProgramMaterials/NodeCalls/Sues%20
Presentation.pdf.
3. Agency for Healthcare Research and Quality. Complicating conditions of
pregnancy and childbirth, 2008. Elixhauser A, Wier L. Healthcare Cost and
Utilization Project (HCUP) Statistical Brief #113. May 2011. Accessed Jun
30, 2015. http://www.ncbi.nlm.nih.gov/books/NBK56037/.
4. Mann S, et al. Assessing quality obstetrical care: Development of standard-
ized measures. Jt Comm J Qual Patient Saf. 2006;32(9):497–505.
5. Nielsen PE, et al. Eects of teamwork training on adverse outcomes and
process of care in labor and delivery: A randomized controlled trial. Obstetr
Gynecol. 2007;109(1):48–55.
6. Balilit J. Measuring the quality of inpatient obstetrical care. Obstet Gynecol
Surv. 2007;62(3):207–213.
7. National Perinatal Information Center: Quality Analytic Services. Accessed
Jun 30, 3025. http://www.npic.org/.
8. Walker S, Strandjord TP, Benedetti TJ. In search of perinatal quality measures:
1 hospital’s in-depth analysis of the Adverse Outcomes Index. Am J Obstet
Gynecol. 2010;203(4):336.e1–7.
9. National Quality Forum. Quality Positioning System (QPS): Measure De-
scription Display Information: 0716 Healthy Term Newborn. (Updated: Oct
13, 2011.) Accessed Jun 30, 2015. http://www.qualityforum.org/QPS/Measure
Details.aspx?standardID=171&print=0&entityTypeID=1.
10. Centers for Medicare & Medicaid Services. Clinical Quality Measures Fi-
nalized for Eligible Hospitals and Critical Access Hospitals Beginning with
FY 2014. Accessed Jun 30, 2015. http://cms.gov/regulations-and-guidance
/legislation/ehrincentiveprograms/downloads/tableof2014_eh_measure
_versions.pdf.
11. California Maternal Quality Care Collaborative. Unexpected Complica-
tions in Term Newborns. Accessed Jun 30, 2015. https://www.cmqcc.org/focus
-areas/quality-metrics/unexpected-complications-term-newborns.
12. Clark EA, Silver RM. Long-term morbidity associated with repeat cesarean
delivery. Am J Obstet Gynecol. 2011;205(6 Suppl):S2–10.
13. Cook JR, et al. Multiple repeat caesarean section in the UK: Incidence
and consequences to mother and child. A national, prospective, cohort study.
BJOG 2013;120(1):85–91.
14. McNiven P, Kaufman K, Enkin M. Measuring birth outcomes: Validating
the perinatal outcome index. Canadian Journal of Midwifery Research and Prac-
tice. 2002;1(2):9–14.
15. Murphy PA, Fullerton JT. Development of the Optimality Index as a new
approach to evaluating outcomes of maternity care. J Obstet Gynecol Neonatal
Nurs. 2006;35(6):770–778
Online Only Content
http://www.ingentaconnect.com/content/jcaho/jcjqs
See the online version of this article for
Appendix 1. Data Collection Worksheet for Quality Outcomes Before
and After TeamSTEPPS® in Labor & Delivery (L&D)
Back-to-back updates on Joint Commission standards,
CMS CoPs, and performance improvement!
To register, visit jcrinc.com or call Customer Ser vice at 877.223.6866.
Joint Commission Resources, Inc. (JCR), a wholly controlled, not-for-profi t a liate of The Joint Commission, is the o cial publisher and educator
of The Joint Commission. JCR is an expert resource for health care organizations, providing consulting services, educational services, publications
and software to assist in improving quality and safety and to help in meeting the accreditation standards of The Joint Commission. JCR provides
consulting services independently from The Joint Commission and in a fully confi dential manner. Please visit www.jcrinc.com for more information.
Hospital Hospital
Executive Briefi ngs CMS Update
Los Angeles, CA September 10, 2015 September 11, 2015
Chicago, IL September 24, 2015 September 25, 2015
New York, NY October 5, 2015 October 6, 2015
Come to Chicago one day early for CJCP® Essentials Prep
on September 23!
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015 AP1
Online Only Content
Appendix 1. Data Collection Worksheet for Quality Outcomes Before and After TeamSTEPPS®
in Labor & Delivery (L&D)
Audit Number (Study Code):
Admit Date Gestational Age
(Wks + Days)
Delivery Date Birth Weight
(Grams)
Disc harge Date
(MM/DD/YYYY)
(MM/DD/YYYY)
(MM/DD/YYYY)
Apgar @1 min
Delivery Meth od Apgar @5 min
Presen t Source Inclusion Cri teria (ICD -9-C M/DRG) Exclusion
Criteria
1. Maternal D eath
DRGs 370–375
Discharge Disposition = De ath
Excludes:
o
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Death after discharge or
transfer out
oTransfer in with delivery at
other loca tion
□
Yes
□
No
□
NPIC
□
Chart
□
Both
□
DRG 370: Cesarean birth with complications or comorbidities
□
DRG 371: Cesarean birth without complications or comorbidities
□
DRG 372: Vaginal Delivery with complications or comorbidities
□
DRG 373: Vaginal Delivery without complications or
comorbidities
□
DRG 374: Vaginal Delivery with sterilization or D&C
□
DRG 375: Vaginal Delivery with OR procedure except
sterilization and/or D&C
Death after
discharge or
transfer out
Transfer in with
delivery at other
location
2. Intra partum or n eona tal
death > 2,500g *
Birth weight ≥ 2,500 grams
Inborn only
Death within 7 days of birth
Excludes:
o Congenital anomalies
(Diagnosis codes 740–759.9)
oFetal hydrops (Diagnosis
code 778.0)
o
o
Death > 7 days
Birth weight < 2,500 grams
□
Yes
□
No
□
NPIC
□
Chart
□
Both
□
740–759.9 = comprehensive range of all ICD-9-CM congenital
anomaly codes available _______ (write in diagnosis code)
□
778.0 = idiopathic fetal hydrops (not due to isoimmunization)
Congenital
anomalies
(Diagnosis
codes 740–
759.9)
Fetal hydrops
(Diagnosis code
778.0)
Death > 7 days
Birth weight <
2,500 grams
3. Uter ine ru pture
Diagnosis cod e 665.1 in
prima ry, rst or second
diagnosis code position only
Excludes D RG 665.1 in ≥ 3
diagnosi s code position
□
Yes
□
No
□
NPIC
□
Chart
□
Both
□
665.1 = rupture of uterus during labor
□
□
□
DRGs 370–375 (as above) _______ (write in diagnosis code)
Diagnosis code
665.1 in
diagnosis code
position
4. Materna l adm ission t o
ICU
DRG 370–375
ICU day or charge at any time
during stay (prior to or after
delivery)
Diagnosis code 640.xx–677.xx
with 5
th
digit = 2 (delivered with
mention of postpartum condition)
Excludes ICU stay or charges
without diagnosis code 640.xx–
677.xx and 5
th
digit = 2
□
Yes
□
No
□
NPIC
□
Chart
□
Both
640.xx–677.xx = includes complications and comorbidities
including hemorrhage, hypertension, infections of genitourinary
tract, other infections, anesthetic complications, other conditions in
mother complicating postpartum condition including diabetes,
thyroid dysfunction, anemia, drug dependence, mental disorders,
congenital cardiovascular disorders, etc.
DRGs 370–375 (as above) _______ (write in diagnosis code)
ICU stay or
charges without
diagnosis code
640.xx–677.xx
and 5
th
digit = 2
5. Birt h traum a
Inborn only (all gestational ages
included)
Diagnosis codes 767.0, 767.11,
767.3, 767.4, 767.5, 767.6,
767.7
Excludes outborn transferred in
□
Yes
□
No
□
NPIC
□
Chart
□
Both
□
767.0 = subdural and cerebral hemorrhage
□
767.11 = epicranial subaponeurotic hemorrhage - massive
□
767.3 = other injuries to the skeleton
□
767.4 = injury to spine and spinal column
□
767.5 = facial nerve injury
□
767.6 = injury to brachial plexus
□
767.7 = other cranial and peripheral nerve injury
Outborn
transferred in
6. Matern al Return to OR /
labor & delivery
DRG 370–375
Any of the following procedure
codes in rst or second position:
75.92, 69.02, 54 .61, 38.86,
39.98, 69.52
Excludes any other procedures
and inclusion list of procedure
codes but in ≥ 3 position
□
Yes
□
No
□
NPIC
□
Chart
□
Both
□
75.92 = evacuation of other hematoma of vulva or vagina ≤
□
69.02 = D&C following delivery
□
54.61 = reclosure of postoperative disruption of the abdominal wall
□
38.86 = other surgical occlusion of abdominal vessels
□
39.98 = control of hemorrhage
□
69.52 = aspiration curettage following delivery
□
DRG 370–375 (as above) _______ (write in diagnosis code)
Any other
procedures and
inclusion list of
procedure c odes
but in ≥ 3
rd
position
□
□
□
□
□
□
□
□
□
□
≥ 3
rd
rd
rd
Adverse Outcome Indicator:
(continued on page AP2)
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015
AP2
Online Only Content
Appendix 1. Data Collection Worksheet for Quality Outcomes Before and After TeamSTEPPS®
in Labor & Delivery (L&D) (continued)
Presen t Source Inclusion Cri teria (ICD -9-C M/DRG) Exclusion
Criteria
7. Admissio n to NICU
> 2,500 g & for > 24
hours †
Inborns only
≥
2,500 grams
≥
37 weeks gestation
ICU charge or LOS > 1 day
Within 1 day of birth
Excludes:
oCongenital anomalies: DRG
740–759.9
oFetal hydrops DRG 778.0
□
Yes
□
No
□
Yes
□
No
□
Yes
□
No
□
Yes
□
No
□
Yes
□
No
□
NPIC
□
Chart
□
Both
□
740–759.9 = comprehensive range of all ICD-9-CM congenital
anomaly codes available
□
778.0 = idiopathic fetal hydrop s (not due to isoimmunization
Congenital
anomalies:
diagnosis codes
740–759. 9
Fetal hydrops:
diagnosis code
778.0
8. APGA R < 7 at 5 minutes
Inborns only and
Birth weight
≥
2,500 grams
Excludes congenital anomalies:
DRG 740–759.9
Excludes fetal hydrops DRG
778.0
□
NPIC
□
Chart
□
Both
□
740–759.9 = comprehensive range of all ICD-9-CM congenital
anomaly codes available
□
778.0 = idiopathic fetal hydrop s (not due to isoimmunization
Congenital
anomalies:
diagnosis codes
740–759.9
Fetal hydrops:
diagnosis code
778.0
9. Maternal B lood
transfu sion
Procedu re code 99.0 or Blood
Transfusion in dicator = 1
Excludes transfers into hospital
after delivery
□
NPIC
□
Chart
□
Both
□
99.0 = transfusion of blood and blood components
□
DRG 370–375 (as above) _______ (write in diagnosis code)
Transfers into
hospital after
delivery
10.3º- or 4 º-perineal tear
DRG 370–375
Diagnosis code 664.2 or 664.3
Excludes transfers into hospital
after delivery
□
NPIC
□
Chart
□
Both
□
664.2 = 3
rd
degree perineal laceration
□
664.3 = 4
th
degree perineal laceration
□
DRG 370–375 (as above) _______ (write in diagnosis code)
Transfers into
hospital after
delivery
Injury
Comments:
Chart r evie w adv erse
outcome consiste nt wi th
AOI a dmini strat ion d ata
algorithm i.e.
Is the iden tified Adve rse
Outco me co nfirmed p er
algorithm?
Is th e ICD9 code accurate
accor ding to definitio n?
□
Y
es
□
No
□
N/A
□
Yes
□
No
□
N/A
□
Yes
□
No
□
N/A
Comments:
Risk fa ctors /pre- existing
conditions relevan t to
adverse outco me
□
Y
es
□
No
□
N/A
Comments:
Chart r evie w adv erse
outcome consiste nt wi th
origi nally conc eived
adverse outco me i.e.
Is the Adver se Ou tcome
ident ified clin ically
relevan t and appropria te?
□
Y
es
□
No
□
N/A
Comments:
Other Comments (use back or attach sheet if needed):
Chart Reviewer’s Name:
Audit Date (MM/DD/YYYY)
□
□
□
□
□
□
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Adverse Outcome Indicator:
Copyright 2015 The Joint Commission