ArticlePDF Available

Accuracy of the Adverse Outcome Index: An Obstetrical Quality Measure

Authors:

Abstract and Figures

In obstetrics, a nationally accepted set of quality indicators for patient safety was not available in the United States until the development of a set of 10 adverse outcome measures-the Adverse Outcome Index (AOI). The National Perinatal Information Center (NPIC) developed hospital discharge data-based algorithms combined with a small set of supplemental patient data for calculation of the AOI. A study was conducted to determine the specificity, sensitivity, positive predictive value (PPV), and negative predictive value (NPV) of the AOI by using the National Perinatal Information Center (NPIC) algorithm. A retrospective chart review of 4,252 obstetrical and neonatal charts from 2003 through 2007 was performed. NPIC definitions were compared with the "gold standard"-chart review. A total of 229 deliveries among the 4,000 randomly selected charts had at least one adverse outcome, reflecting an AOI of 5.7%. For detection of the 10 adverse outcomes within the AOI, the overall sensitivity of the AOI was 81.7%, specificity was 98.2%, PPV was 86.3%, and NPV was 97.4%. The Kappa value for agreement between the coded charts and the chart review was 0.82 (standard deviation = 0.01, 95% confidence interval [CI] = 0.80-0.85), which is considered very good. The AOI is highly reliant on accurate coding and provider documentation and requires validation with manual chart review. Concurrent chart review improves the accuracy of the AOI. Caution is advised when using the AOI as an exclusive measure of assessing obstetric quality because it may be heavily influenced by a single outcome measure; perineal laceration rates represented twice the frequency of all other outcomes combined. The AOI should be modified to better measure preventable adverse events and include a means of accounting for preexisting conditions.
Content may be subject to copyright.
www.jcrinc.com
Improvement from
Front Oce to Front Line
August 2015
Volume 41 Number 8
Computerized Provider
Order Entry as a
Cost-Savings Strategy
Features
Information Technology
e Value of Computerized Provider Order Entry: Is It Time for
the Debate to Be Over?
Implementing Computerized Provider Order Entry in Acute Care
Hospitals in the United States Could Generate Substantial Savings
to Society
Medication Safety
Advancing Medication Safety: Establishing a National Action Plan
for Adverse Drug Event Prevention
Implementation Science
Facilitation of a Multihospital Community of Practice to
Increase Enrollment in the Hospital to Home National Quality
Improvement Initiative
Performance Measures
Accuracy of the Adverse Outcome Index: An Obstetrical
Quality Measure
Departments
Forum
Connecting Patients and Clinicians: e Anticipated Eects
of Open Notes on Patient Safety and Quality of Care
“The probability that
implementing CPOE in acute
care hospitals would, on
average, and using typical
estimates of implementation
costs, improve health and
yield cost savings to society
exceeds 99%.
—Implementing Computerized Provider
Order Entry in Acute Care Hospitals in
the United States Could Generate
Substantial Savings to Society
(p. 341)
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015
370
Pregnancy and childbirth are the leading causes of hospital
admissions for women in the United States. In 2009 “live
born infant” was the most commonly coded diagnosis, account-
ing for more than 10% of all hospital stays.1 In 2012 the Insti-
tute for Healthcare Improvement (Cambridge, Massachusetts)
reported that obstetrical adverse events of some sort occur in
9% of all deliveries.2 Deliveries with complications are costly;
in 2008 they accounted for $17.4 billion, which represents 5%
of total hospital costs in the United States for that year. A hospi-
tal stay for a pregnancy with complications is about 50% more
costly than one without complications.3
In obstetrics, a nationally accepted set of quality indicators
for patient safety was not available until a multidisciplinary
panel of experts met in June 2001 in an attempt to develop
an acceptable set of measures.4 Professional participants in the
subsequent consensus conferences developed a set of 10 adverse
outcome measures, called the Adverse Outcome Index (AOI).
e US Department of Defense and the Harvard Risk Man-
agement Strategies Foundation (Cambridge) rst examined the
impact of team training and teamwork on obstetrical care and
the AOI in a multicenter labor and delivery study involving 15
institutions and more than 28,000 patients, in 2002.5 Subse-
quently, further evaluation of the AOI as a proposed inpatient
quality measure was advocated.6 e National Perinatal Infor-
mation Center (NPIC; Providence, Rhode Island) developed
hospital discharge data–based algorithms combined with a small
set of supplemental patient data for calculation of the AOI. e
NPIC administrative algorithm is an index of 10 adverse out-
comes, obtained retrospectively from coded discharge data. is
generates a list of births with one or more adverse outcomes,
with a designated list of International Classication of Diseases,
Ninth Revision, Clinical Modication (ICD-9-CM)/Diagnosis-
Related Group (DRG) as inclusion and exclusion criteria.
NPIC uses the AOI to track adverse obstetrical outcomes na-
tionwide.7 To date, only one study evaluated the AOI as derived
from the NPIC algorithms. e nding of variable accuracy for
hospital perinatal adverse event measurement in this small-N
Performance Measures
Accuracy of the Adverse Outcome Index: An Obstetrical
Quality Measure
Lisa M. Foglia, MD; Peter E. Nielsen, MD; Eileen A. Hemann, EdD; Suzan Walker, MPH; Jason A. Pates, MD;
Peter G. Napolitano, MD; Shad Deering, MD
Article-at-a-Glance
Background: In obstetrics, a nationally accepted set of
quality indicators for patient safety was not available in the
United States until the development of a set of 10 adverse
outcome measures—the Adverse Outcome Index (AOI).
e National Perinatal Information Center (NPIC) devel-
oped hospital discharge data–based algorithms combined
with a small set of supplemental patient data for calculation
of the AOI. A study was conducted to determine the speci-
city, sensitivity, positive predictive value (PPV), and nega-
tive predictive value (NPV) of the AOI by using the National
Perinatal Information Center (NPIC) algorithm.
Methods: A retrospective chart review of 4,252 obstetri-
cal and neonatal charts from 2003 through 2007 was per-
formed. NPIC denitions were compared with the “gold
standard”—chart review.
Results: A total of 229 deliveries among the 4,000 random-
ly selected charts had at least one adverse outcome, reecting
an AOI of 5.7%. For detection of the 10 adverse outcomes
within the AOI, the overall sensitivity of the AOI was 81.7%,
specicity was 98.2%, PPV was 86.3%, and NPV was 97.4%.
e Kappa value for agreement between the coded charts and
the chart review was 0.82 (standard deviation = 0.01, 95%
condence interval [CI] = 0.80–0.85), which is considered
very good.
Discussion: e AOI is highly reliant on accurate coding
and provider documentation and requires validation with
manual chart review. Concurrent chart review improves the
accuracy of the AOI. Caution is advised when using the AOI
as an exclusive measure of assessing obstetric quality because
it may be heavily inuenced by a single outcome measure;
perineal laceration rates represented twice the frequency of
all other outcomes combined. e AOI should be modied
to better measure preventable adverse events and include a
means of accounting for preexisting conditions.
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015 371
(33 mother/infant pairs) study led the authors to advocate for
better understanding and evaluation of the individual metrics.8
e purpose of the current study was to determine the specici-
ty, sensitivity, positive predictive value (PPV), and negative pre-
dictive value (NPV) of the AOI as it is currently calculated on
a larger sample and determine its accuracy for our institution.
Methods
Setting
is study was performed in 2011 at a tertiary care military
health care facility in the Pacic Northwest, with an average
delivery rate of 200 births per month. is was a retro spective
chart review of selected delivery and associated birth records
from 2003 through 2007. e charts were reviewed for adverse
outcomes using the NPIC AOI denitions, with their respec-
tive inclusion and exclusion criteria.
Manual Chart review
e AOI assesses a dichotomous state (such as the presence
or absence of an adverse event), when the true state is known
by some “gold standard.” In this study, the gold standard was
a manual chart review of the patient’s electronic health record
(EHR) by one of four board-certied obstetrician/gynecolo-
gists [L.M.F., J.A.P., P.G.N., S.D.]. Chart review ndings were
compared to the adverse outcomes identied using the NPIC
administrative algorithm. We determined the sensitivity, spec-
icity, PPV, and NPV of the AOI for each adverse outcome.
SaMple and proCeSS for Chart review
On the basis of a sensitivity of the computer-determined AOI
relative to chart-reviewed AOI of 90%, with a 95% condence
interval [CI] precision +/– 5%, 4,000 charts were needed for
this review. Charts were randomly selected from all births in the
study period, using the Excel™ randomization function (rand-
between, 1,7539; Microsoft™, Redmond, California), which
assigns a unique number to each hospital birth. e list was
sorted in ascending order, and the rst 2,000 births were select-
ed for chart review for each of the two time periods. We also in-
cluded 252 charts from this time period that were identied by
the NPIC algorithm as having an adverse event. is resulted in
a total of 4,252 charts for review (Figure 1, above) We elected to
include the additional 252 charts with at least one adverse out-
come identied by the NPIC algorithm to evaluate the accuracy
of all records with adverse outcomes identied by NPIC. Charts
(History and Physical, Delivery Summary, and Discharge Sum-
mary) were reviewed to verify correct identication of one or
Figure 1. All hospital births from two designated time periods were randomized to yield 2,000 mother/infant charts for a total of 4,000 charts. After this list
was obtained, it was compared with the Adverse Outcome Index (AOI) obtained from the National Perinatal Information Center (NPIC) for the same time
period. Any births from the NPIC list (AOI present, when an administrative algorithm was used) that were not selected in the randomized samples were added
to the chart review, yielding 4,252 mother/infant records to be reviewed for adverse outcomes.
Selection of Study Population
7,539 Hospital
deliveries during
designated study time
periods
3,530 deliveries between
10/01/01–09/30/03
randomized to yield
2,000 births charts to be
reviewed for AOIs
4,009 deliveries between
01/01/05–12/31/06
randomized to yield
2,000 births charts to be
reviewed for AOIs
7,539 Hospital
deliveries during
designated study time
periods
3,530 deliveries between
10/01/01 and 09/30/03
randomized to yield
2,000 birth charts to be
reviewed for AOIs
4,009 deliveries between
01/01/05 and 12/31/06
randomized to yield
2,000 birth charts to be
reviewed for AOIs
503 NPIC deliveries
with AOIs using
the administrative
algorithm
252 deliveries not
included in randomized
birth lists when
compared with NPIC
AOI cases
503 NPIC deliveries
with AOIs using
the administrative
algorithm
252 deliveries not
included in randomized
birth lists when
compared with NPIC
AOI cases
AOI found were
validated by
Obstetrical Physician
4,252 total cases
reviewed for AOIs
AOIs found were
validated by
Obstetrical Physician
4,252 total cases
reviewed for AOIs
++=
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015
372
more of the 10 adverse outcomes in the AOI compared with
the NPIC algorithm–identied cases. Additional notes were re-
viewed, as necessary to verify or dispute the information iden-
tied by NPIC (for example, pre- and posttransfusion notes for
patients receiving blood products). e data collection sheet is
provided in Appendix 1 (available in online article). e PPV
and NPV were calculated for the NPIC administrative algo-
rithm–identied adverse outcomes. is study was approved by
the hospital Institutional Review Board, and a waiver of consent
was obtained.
Results
Maternal delivery and neonatal reCordS
We reviewed 4,252 delivery and neonatal records. e overall
sensitivity of the AOI was 81.7%, PPV was 86.3%, specicity
was 98.2%, and NPV was 97.4%. Patients with one or more
adverse outcome were more likely to have larger birth weights,
higher rates of operative vaginal delivery, and lower rates of
cesarean delivery (Table 1, above).
adverSe outCoMeS
A total of 530 adverse outcomes occurred among 481 de-
liveries in the 4,252 records reviewed. A total of 229 deliveries
among the 4,000 randomly selected charts had at least one ad-
verse outcome, reecting an AOI of 5.7% within this random
sample of records. Four hundred forty-one deliveries had one
adverse event, 33 deliveries had two adverse events, 6 deliveries
had three adverse events, and 1 delivery had ve adverse events.
e most frequent adverse event was 3rd or 4th degree lacera-
tion, followed by maternal blood transfusion, birth trauma, in-
fant admission to the neonatal ICU (NICU), maternal admit to
ICU, ve-minute Apgar of < 7, maternal return to the operating
room, uterine rupture, and term neonatal death. ere were no
maternal deaths per the chart review or NPIC algorithm during
the study period (Figure 2, page 374; color version available in
online article).
e Kappa value for agreement between the coded charts
and the chart reviews, as calculated using GraphPad (http://
www.graphpad.com/quickcalcs/kappa1.cmf), was 0.82 (stan-
dard deviation = 0.01, 95% CI = 0.80–0.85). Table 2 (page
373) lists the strength of agreement of the indicators. Measures
of accuracy were calculated for 9 of the 10 adverse outcomes
(maternal death excluded). Both term neonatal deaths were cor-
rectly identied using the algorithm and conrmed by chart
review. e algorithm had a sensitivity of 80% for three out-
comes: 3rd or 4th degree laceration, uterine rupture, and birth
trauma. e algorithm had the lowest sensitivity (≤ 50%) for
maternal return to the operating room, maternal blood trans-
fusions, maternal admission to the ICU, infant admission to
NICU, and a ve-minute Apgar score of < 7. e specicity
of the algorithm was greater than 99% for all adverse outcome
indicators (Table 2). e NPV was greater than 90% for all
indicators. However, the PPV of the algorithm for infant ad-
missions to the NICU, ve-minute Apgar scores of < 7, ma-
ternal return to the operating room, and uterine rupture was
less than 70%. e adverse outcome indicators with the highest
PPV were term neonatal death, maternal admissions to ICU,
maternal blood transfusion, 3rd or 4th degree laceration, and
birth trauma (Table 2).
Discussion
e nding of this study supports the wide variability in accu-
racy of the PPVs for the AOI among the 10 adverse outcomes,
as found by Walker, Strandjord, and Bennedetti.8 eir study
identied a PPV of 0 for term neonatal death due to incor-
rect birth weight codes, as well as relatively high PPVs for ve-
Table 1. Characteristics of the Sampled Population
Population with One or More
Adverse Events (N = 481)
Population Without an Adverse
Event (N = 3,771) P Value
Gestation Age-weeks (mean + SD) 38.7 + 2.7 38.6 + 2.6 .476
Birth Weight-grams (mean + SD) 3,431 + 730 3,331 + 641 .003
Method of Delivery Frequency Percentage Frequency Percentage
Spontaneous Vaginal 222 46.2 2,632 69.8 < .001
Forceps 145 30.1 101 2.7 < .001
Vacuum 47 9.8 85 2.3 < .001
Cesarean 67 13.9 953 25.3 < .001
Total 481 100.0 3,771 100.0
SD, standard deviation.
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015 373
minute Apgar scores and NICU admission—which were due
to improved methods to document Apgar scores and chang-
es in NICU billing codes. In our study, the most accurately
identied indicators were term neonatal death, uterine rup-
ture, birth trauma, and 3rd or 4th degree laceration. e least
accurately identied events were term infants admitted to the
NICU, ve-minute Apgar scores of < 7, and maternal return to
the operating room. Because there were no maternal deaths in
our population, the accuracy of this indictor could not be de-
termined. Maternal admission to the ICU, NICU admissions,
and identication of uterine rupture, which heavily relied on
provider documentation and the resultant codes used for diag-
noses and billing, were the most dicult to accurately identify
via the algorithm.
Examining the list of codes in the NPIC algorithm for com-
prehensiveness is warranted. For example, birth trauma codes
included in the NPIC denition are subdural and cerebral
hemorrhage (767.00), epicranial subaponeurotic hemorrhage
(767.11), other injuries to the skeleton (767.30), injury to spine
and spinal column (767.40), facial nerve injury (767.50), inju-
ry to brachial plexus (767.60), and other cranial and periph-
eral nerve injury (767.70). Not included in the algorithm but
identied in our chart review were clavicular fracture (767.20)
and right hip dislocation from breech delivery (754.32), both
of which were documented as birth trauma. Adding these diag-
noses to the list captured by the AOI for this adverse outcome
should be considered.
e ability to distinguish between a maternal return to
the operating room to address a complication versus an inde-
pendent second procedure performed immediately following
delivery (for example, postpartum bilateral tubal ligation) was
challenging for both the NPIC algorithm and chart review.
Medical record reviews proved time consuming for this measure
because the documentation was found in text elds, as opposed
to data elements, which could be queried.
System-specic issues can also aect the accuracy of the AOI,
which further indicated the importance of concomitant medical
record reviews to identify areas aecting the AOI accuracy. For
example, neonatal admissions to the NICU are subject to hospi-
tal-specic admission policies and criteria. At our institution, it
was dicult to identify admission to the NICU versus interme-
diate care nursery because they are physically located in the same
place and are both indicated as “NICU” in the EHR, which can
overinate the NICU admissions identied by the algorithm.
For example, an infant admitted for observation because of ma-
ternal chorioamnionitis appeared as a NICU admission in the
NPIC algorithm. Although these infants required a higher level
of care than routine postpartum care, they were not of the acuity
level required for a NICU admission and were documented as
such on manual chart review. Revising this discrete-level indica-
tor to a severity ordinal scale with a cuto threshold to include
only the most severe admission diagnoses to the NICU should
be considered to rene the accuracy of this outcome.
In 2011 the National Quality Forum endorsed a new quality
measure, Healthy Term Newborn, stewarded by the California
Maternal Quality Care Collaborative.9 is measure, a compos-
ite of unexpected neonatal complications that developed during
birth or neonatal care, is included in the 29–clinical quality
Table 2. Characteristics of the National Perinatal Information Center Adverse Outcome Index Algorithm
Compared to Chart Audit (N = 4,252)
Adverse Outcome Sensitivity (%) Specicity (%) PPV (%) NPV (%) Kappa Coefcient
Kappa
Strength of
Agreement* 95% CI SE
Maternal Death*
Neonatal Death 100.0 100.0 100.0 100.0 1.00 Perfect 1.00–1.00 0.00
3rd or 4th Degree
Laceration 98.0 99.6 95.2 99.8 0.96 Very Good 0.95–0.98 0.01
Birth Trauma 83.7 99.8 80.4 99.8 0.82 Very Good 0.74–0.90 0.04
Uterine Rupture 100.0 99.9 66.7 100.0 0.80 Very Good 0.53–1.07 0.14
Maternal Admission to ICU 47.4 100.0 100.0 99.8 0.64 Good 0.44–0.85 0.11
Apgar < 7 at 5 minutes 26.3 99.8 38.5 99.7 0.31 Fair 0.11–0.52 0.11
Infant Admission to NICU 30.0 99.4 25.0 99.5 0.27 Fair 0.13–0.41 0.07
PPV, positive predictive value; NPV, negative predictive value; CI, condence interval; SE, standard error.
* Poor = < 2.0; fair = .20–<.40; moderate = .40–< .60; good = .60–< .80; very good = .80–1.00.
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015
374
Figure 2. Charts (history and physical, delivery summary, and discharge summary; “Gold Standard/Electronic Health Record [EHR] Chart Review”)
were reviewed to verify correct identication of 1 or more of the 10 adverse outcomes in the Adverse Outcome Index compared with the National Perinatal
Information Center (NPIC) algorithm–identied cases. Positive predictive values (PPVs), negative predictive value (NPVs) and subsequent specicity and
sensitivity values for detecting the 10 adverse outcomes are shown. e overall sensitivity of the AOI was 81.7%, specicity was 98.2%, PPV was 86.3%, and
NPV was 97.4%. TP, true positive; FP, false positive; TN, true negative; FN, false negative; OR/L&D, operating room/labor and delivery.
Specicity and Sensitivity Values for 4,252 Delivery and Neonatal Records
Positive Negative
Positive 433 69
PPV
86.25%
TP/(TP + FP)
percentage
Negative 97 3,653
NPV
97.41%
TN/(FN + TN)
percentage
Sensitivity Specificity
81.70%98.15%
TP/(TP + FN) TN/(FP + T N)
percentagepercentage
Positive Negative Positive Negative
Positive 0 0
PPV
N/A
Positive 4 4
PPV
50.00%
Negative 0 4,252
NPV
100.00%
Negative 4 4,240
NPV
99.91%
Sensitivity Specificity Sensitivity Specificity
0.00%100.00%50.00% 99.91%
Positive Negative Positive Negative
Positive 2 0
PPV
100.00%
Positive 9 27
PPV
25.00%
Negative 0 4,250
NPV
100.00%
Negative 21 4,195
NPV
99.50%
Sensitivity Specificity Sensitivity Specificity
0.00%100.00%30.00%99.36%
Positive Negative Positive Negative
Positive 4 2
PPV
66.67%
Positive 5 8
PPV
38.46%
Negative 0 4,246
NPV
100.00%
Negative 14 4,225
NPV
99.67%
Sensitivity Specificity Sensitivity Specificity
100.00%99.95%26.32%99.81%
Positive Negative Positive Negative
Positive 9 0
PPV
100.00%
Positive 25 1
PPV
96.15%
Negative 10 4,233
NPV
99.76%
Negative 33 4,193
NPV
99.22%
Sensitivity Specificity Sensitivity Specificity
47.37%100.00%43.10%99.98%
Positive Negative Positive Negative
Positive 41 10
PPV
80.39%
Positive 334 17
PPV
95.16%
Negative 8 4,193
NPV
99.81%
Negative 7 3,894
NPV
99.82%
Sensitivity Specificity Sensitivity Specificity
83.67%99.76%97.95%99.57%
NPIC Administrative
Algorithm
Infant Admit to NICU
Gold Standard/EHR
Chart Review
NPIC Administrative
Algorithm
3rd/4th Degree
Lacerations
Gold Standard/EHR
Chart Review
Apgars < 7 @ 5 minutes
Gold Standard/EHR
Chart Review
NPIC Administrative
Algorithm
Maternal Blood
Transfusions
Gold Standard/EHR
Chart Review
Maternal Return to
OR/L&D
Gold Standard/EHR
Chart Review
NPIC Administrative
Algorithm
NPIC Administrative
Algorithm
Neonatal Death
Gold Standard/EHR
NPIC Administrative
Algorithm
Uterine Rupture
Gold Standard/EHR
NPIC Administrative
Algorithm
Maternal Admit to ICU
Gold Standard/EHR
NPIC Administrative
Algorithm
Birth Trauma
Gold Standard/EHR
NPIC Administrative
Algorithm
NPIC Administrative
Algorithm
Overall
Gold Standard/EHR
NPIC Administrative
Algorithm
Maternal Death
Gold Standard/EHR
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015 375
measure set for Stage 2 of the Center for Medicare & Medicaid
Services EHR incentive program.10 Although not risk adjusted,
the measure does exclude preexisting conditions, such as multi-
ple gestations, preterm birth, congenital anomalies, growth re-
striction, and fetuses aected by selected maternal conditions.
In addition, the numerator straties the severity of complica-
tions. e severe category refers to death, a ve-minute Apgar
score of ≤ 3, and diagnosis codes for major birth injuries or
major neurologic, pulmonary, or infectious complications. e
moderate category refers to those with lesser diagnoses resulting
in longer lengths of stay but a low likelihood of medium, or
long-term morbidity, such as transient tachypnea of the new-
born. Feedback to the measure developers resulted in an update
to the measure to change the numerator to capture “Unex-
pected Newborn Complications” in those newborns without
preexisting complications who experience a severe or moderate
morbidity.11 is measure is complementary to the AOI, as it
provides a more precise neonatal outcome metric. Alternately,
this measure could replace components of the AOI (ve-minute
Apgar scores of < 7, birth trauma and infant admission to the
NICU) because it includes all these components, straties birth
injuries by severity, and excludes preexisting conditions.
Coding did not capture Apgar scores of < 7 at ve minutes
because codes do not exist for low Apgar scores; identication
required a manual chart review. With the national emphasis on
EHRs, collection of prospective data about obstetric outcomes
should be considered. Maternal blood transfusion was another
indicator aected by the maturity of our coding system. Before
June 2006, blood transfusions were not coded in discharge pa-
perwork at all, accounting for some of the loss of accuracy in
this indicator. After this date, there was a high correlation be-
tween the accuracy of the coding and the chart audit. Data for
sensitivity, specicity, PPV, and NPV before and after the cod-
ing change are shown in Figure 3 (above, right; color version
available in online article). e inclusion and exclusion criteria
for this indictor may also need to be reconsidered with a time
threshold from delivery. Blood transfusions associated with the
most serious adverse outcomes were identied in conjunction
with admission to the operating room or admission to ICU
within the rst 24 hours after delivery. Transfusions after the
rst 24 hours were primarily for symptomatic anemia.
Finally, the largest total number of adverse outcomes was
3rd or 4th degree laceration. Identication relied heavily on
the coder’s interpretation of physician documentation. Lacer-
ations were primarily mismatched when the physician docu-
mented a 2nd degree laceration with extension into the rectal
capsule (“partial 3rd degree laceration”) and the coding was for
a second-degree laceration instead of a 3rd degree laceration,
which encompasses a wide variation in the amount of sphinc-
ter damage (partial disruption of the external anal sphincter to
complete disruption of the internal and external anal sphinc-
ter). ese may be of varying clinical signicance with regard to
adverse outcomes.
Because perineal lacerations are associated exclusively with
vaginal delivery, interpretation of a high AOI may lead to erro-
neous conclusions that vaginal deliveries are less desirable than
cesarean deliveries. A strategy to lower the AOI by increasing
cesarean deliveries fails to account for the short- and long-term
risks associated with future pregnancies, such as uterine rupture
and placental implantation abnormalities, and their attendant
increased risk of maternal blood transfusion, hysterectomy, ICU
admission, and maternal death.12,13 erefore, the AOI should
be used in combination with other quality indicators that ac-
knowledge dierences in the types, rates, and severity of adverse
outcomes, depending on the mode of delivery. is will help
avoid inadvertently driving practice patterns toward increasing
Figure 3. Before June 2006, blood transfusions were not coded in discharge
paperwork at all, accounting for some of the loss of accuracy in this indicator.
After June 2006, there was a high correlation between the accuracy of
the coding and the chart review. Data for sensitivity, specicity, positive
predictive value (PPV), and negative predictive value (NPV) before and after
the coding change are shown for the National Perinatal Information Center
(NPIC) algorithm–identied cases. EHR, electronic health record; TP, true
positive; FP, false positive; TN, true negative; FN, false negative.
Maternal Blood Transfusion Sensitivity and
Specicity Before and After 2006
Positive Negative
Positive 1 1
PPV
50.00%
TP/(TP = FP)
percentage
Negative 17 3,552
NPV
99.60%
TN/(FN + TN)
percentage
Sensitivity Specificity
5.56%99.98%
TP/(TP + FN) TN/(FP + TN)
percentagepercentage
Maternal Blood
Transfusion before
June 2006
Gold Standard/EHR
Chart Review
NPIC
Administrative
Algorithm
Positive Negative
Positive 24 0
PPV
100.00%
TP/(TP = FP)
percentage
Negative 16 641
NPV
99.62%
TN/(FN + TN)
percentage
Sensitivity Specificity
60.00%100.00%
TP/(TP + FN) TN/(FP + TN)
percentagepercentage
Maternal Blood
Transfusion June
2006 and after
Gold Standard/EHR
Chart Review
NPIC
Administrative
Algorithm
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015
376
cesarean delivery rates to reduce rates of 3rd and 4th degree lac-
erations and subsequently reduce an institutions AOI. Alterna-
tively, separate AOIs for vaginal and cesarean deliveries could
be considered because some complications are unique to each
mode of delivery. Finally, because the optimal operative delivery
rate (cesarean and operative vaginal) relative to adverse outcome
rate is not known, additional study is required to determine
what operative delivery rate optimizes clinical outcomes. It is
clear that operative delivery rates of both zero and 100% would
result in unacceptable rates of maternal and/or neonatal mor-
bidity and mortality.
e AOI is signicantly inuenced not only by high-volume
outcomes signicantly aecting the AOI but by events that may
represent appropriate medical care for preexisting and/or latent
or underlying patient conditions. For example, in an anemic pa-
tient undergoing a repeat cesarean delivery, a blood transfusion
may be medically appropriate and not indicative of an untow-
ard medical event. erefore, labeling all blood transfusions as
adverse events might have the unintended consequence of cast-
ing a negative connotation on a medically appropriate interven-
tion. e AOI also fails to account for adverse events that are
preventable by modiable patient lifestyle factors and preexist-
ing maternal and fetal medical/surgical comorbidities—such as
hypertension, maternal obesity, control of gestational/pregesta-
tional diabetes and its eect on birth weight, preexisting anemia,
maternal bleeding disorders, maternal collagen disorders, and
prior cesarean delivery and abnormal placentation—that sub-
stantially contribute to the risk of certain complications, One
of the major criticisms of the AOI is that many of the indicators
need to be risk adjusted for patient case mix.6 Discussions on the
inclusion and exclusion criteria of the index to make the AOI
more clinically relevant and useful are strongly recommended.
When any measurement instrument is used to collect data,
the users must be cognizant of the accuracy of the resulting
data, as assessed by validation. Some outcomes are easily ex-
tracted using coded data, while others require secondary data
sets or chart review by a provider for complicated cases. e
AOI can be used by health care organizations as one of many
tools in their quality arsenal. As with any tool, validation im-
proves the accuracy of the index and identies opportunities
for improvement. A focus on improved documentation in the
medical record through concurrent review of the chart prior to
patient discharge with direct feedback to physicians and nurses
will improve data accuracy. In addition, as adverse outcomes are
rare obstetrical events, several studies have developed indices of
perinatal outcomes, shifting the focus of measurement to nor-
malcy indicators instead of rare obstetrical outcomes.13,14
Summary
e AOI can be used as a quality benchmarking tool, but it has
some limitations. First, it is heavily inuenced by high-volume
outcomes, such as perineal lacerations. As with any outcome
assessment tool, it does not stratify birth injuries by severity or
exclude preexisting conditions that may increase the chances of
an adverse outcome. Also, the AOI is not adjusted to account
for the short- and long-term risks associated with cesarean de-
liveries. Preexisting risks (such as multiple previous cesarean
deliveries with subsequent abnormal placentation) may place
a patient at greater risk for an unpreventable adverse outcome
assessed by the AOI, in comparison to a healthy nullipara. Fi-
nally, it is highly reliant on the expertise of hospital coding and
pro vider documentation and requires validation with a repre-
sentative sam ple of chart audits.
We recommend that health care organizations using NPIC
algorithms undertake data validation using concurrent chart
reviews, as we have reported here, to guide improvement of pa-
tient care and safety. J
The work described in this article was supported by the TRICARE Management
Activity, Falls Church, Virginia. The views expressed herein are those of the authors
and do not reect the ofcial policy of the Department of the Army, the Department
of Defense, or the US Government. The National Perinatal Information Center is
the contract quality measurement vendor for all hospitals in the military healthcare
system.
Lisa M. Foglia, MD, is Residency Program Director, Department
of Obstetrics and Gynecology, Madigan Army Medical Center, Ta-
coma, Washington. Peter E. Nielsen, MD, formerly Chief of Clini-
cal Operations, Western Regional Medical Command, and Obstetrics
and Gynecology Consultant to US Army Surgeon General, Joint
Base Lewis-McChord, Washington, is Commander, General Leon-
ard Wood Army Community Hospital, Fort Leonard Wood, Missouri.
Eileen A. Hemann, EdD, is Perinatal Clinical Outcomes Research
Nurse, Department of Obstetrics and Gynecology, Madigan Army
Medical Center. Suzan Walker, MPH, is Perinatal Database Manager
and Regional Perinatal Network Coordinator, Pediatrics Division of
Neonatology, University of Washington School of Medicine, Seattle.
Jason A. Pates, MD, is Chief, Division of Obstetrics, Department of
Obstetrics and Gynecology, Madigan Army Medical Center. Peter G.
Napolitano, MD, is Program Director, Maternal-Fetal Medicine Fel-
lowship, and Director, Division of Maternal-Fetal Medicine, Depart-
ment of Obstetrics and Gynecology, Madigan Army Medical Center.
Shad Deering, MD, is Assistant Dean of Simulation Education and
Deputy Medical Director, Uniformed Services University of the Health
Sciences Simulation Center, and Chair, US Army Central Simulation
Committee, Uniformed Services of the Health Sciences, Bethesda,
Maryland. Please address correspondence to Peter E. Nielsen,
peter.e.nielsen.mil@mail.mil.
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015 377
References
1. Agency for Health Care Policy and Research. Most Common Diagnoses and
Procedures in U.S. Community Hospitals, 1996. Elixhauser A, Steiner CA.
AHCPR Publication No. 99-0046. Oct 1999. Accessed Jun 30, 2015. http://
www.hcup-us.ahrq.gov/reports/natstats/commdx/commdx.htm.
2. Gullo S. How-to-Guide: Prevent Obstetrical Adverse Events. Cambridge, MA:
Institute for Healthcare Improvement, 2011. Accessed Jun 30, 2015. http://
www.ihi.org/offerings/Documents/ProgramMaterials/NodeCalls/Sues%20
Presentation.pdf.
3. Agency for Healthcare Research and Quality. Complicating conditions of
pregnancy and childbirth, 2008. Elixhauser A, Wier L. Healthcare Cost and
Utilization Project (HCUP) Statistical Brief #113. May 2011. Accessed Jun
30, 2015. http://www.ncbi.nlm.nih.gov/books/NBK56037/.
4. Mann S, et al. Assessing quality obstetrical care: Development of standard-
ized measures. Jt Comm J Qual Patient Saf. 2006;32(9):497–505.
5. Nielsen PE, et al. Eects of teamwork training on adverse outcomes and
process of care in labor and delivery: A randomized controlled trial. Obstetr
Gynecol. 2007;109(1):48–55.
6. Balilit J. Measuring the quality of inpatient obstetrical care. Obstet Gynecol
Surv. 2007;62(3):207–213.
7. National Perinatal Information Center: Quality Analytic Services. Accessed
Jun 30, 3025. http://www.npic.org/.
8. Walker S, Strandjord TP, Benedetti TJ. In search of perinatal quality measures:
1 hospital’s in-depth analysis of the Adverse Outcomes Index. Am J Obstet
Gynecol. 2010;203(4):336.e1–7.
9. National Quality Forum. Quality Positioning System (QPS): Measure De-
scription Display Information: 0716 Healthy Term Newborn. (Updated: Oct
13, 2011.) Accessed Jun 30, 2015. http://www.qualityforum.org/QPS/Measure
Details.aspx?standardID=171&print=0&entityTypeID=1.
10. Centers for Medicare & Medicaid Services. Clinical Quality Measures Fi-
nalized for Eligible Hospitals and Critical Access Hospitals Beginning with
FY 2014. Accessed Jun 30, 2015. http://cms.gov/regulations-and-guidance
/legislation/ehrincentiveprograms/downloads/tableof2014_eh_measure
_versions.pdf.
11. California Maternal Quality Care Collaborative. Unexpected Complica-
tions in Term Newborns. Accessed Jun 30, 2015. https://www.cmqcc.org/focus
-areas/quality-metrics/unexpected-complications-term-newborns.
12. Clark EA, Silver RM. Long-term morbidity associated with repeat cesarean
delivery. Am J Obstet Gynecol. 2011;205(6 Suppl):S2–10.
13. Cook JR, et al. Multiple repeat caesarean section in the UK: Incidence
and consequences to mother and child. A national, prospective, cohort study.
BJOG 2013;120(1):85–91.
14. McNiven P, Kaufman K, Enkin M. Measuring birth outcomes: Validating
the perinatal outcome index. Canadian Journal of Midwifery Research and Prac-
tice. 2002;1(2):9–14.
15. Murphy PA, Fullerton JT. Development of the Optimality Index as a new
approach to evaluating outcomes of maternity care. J Obstet Gynecol Neonatal
Nurs. 2006;35(6):770–778
Online Only Content
http://www.ingentaconnect.com/content/jcaho/jcjqs
See the online version of this article for
Appendix 1. Data Collection Worksheet for Quality Outcomes Before
and After TeamSTEPPS® in Labor & Delivery (L&D)
Back-to-back updates on Joint Commission standards,
CMS CoPs, and performance improvement!
To register, visit jcrinc.com or call Customer Ser vice at 877.223.6866.
Joint Commission Resources, Inc. (JCR), a wholly controlled, not-for-profi t a liate of The Joint Commission, is the o cial publisher and educator
of The Joint Commission. JCR is an expert resource for health care organizations, providing consulting services, educational services, publications
and software to assist in improving quality and safety and to help in meeting the accreditation standards of The Joint Commission. JCR provides
consulting services independently from The Joint Commission and in a fully confi dential manner. Please visit www.jcrinc.com for more information.
Hospital Hospital
Executive Briefi ngs CMS Update
Los Angeles, CA September 10, 2015 September 11, 2015
Chicago, IL September 24, 2015 September 25, 2015
New York, NY October 5, 2015 October 6, 2015
Come to Chicago one day early for CJCP® Essentials Prep
on September 23!
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015 AP1
Online Only Content
Appendix 1. Data Collection Worksheet for Quality Outcomes Before and After TeamSTEPPS®
in Labor & Delivery (L&D)
Audit Number (Study Code):
Admit Date Gestational Age
(Wks + Days)
Delivery Date Birth Weight
(Grams)
Disc harge Date
(MM/DD/YYYY)
(MM/DD/YYYY)
(MM/DD/YYYY)
Apgar @1 min
Delivery Meth od Apgar @5 min
Presen t Source Inclusion Cri teria (ICD -9-C M/DRG) Exclusion
Criteria
1. Maternal D eath
DRGs 370–375
Discharge Disposition = De ath
Excludes:
o
Death after discharge or
transfer out
oTransfer in with delivery at
other loca tion
Yes
No
NPIC
Chart
Both
DRG 370: Cesarean birth with complications or comorbidities
DRG 371: Cesarean birth without complications or comorbidities
DRG 372: Vaginal Delivery with complications or comorbidities
DRG 373: Vaginal Delivery without complications or
comorbidities
DRG 374: Vaginal Delivery with sterilization or D&C
DRG 375: Vaginal Delivery with OR procedure except
sterilization and/or D&C
Death after
discharge or
transfer out
Transfer in with
delivery at other
location
2. Intra partum or n eona tal
death > 2,500g *
Birth weight2,500 grams
Inborn only
Death within 7 days of birth
Excludes:
o Congenital anomalies
(Diagnosis codes 740–759.9)
oFetal hydrops (Diagnosis
code 778.0)
o
o
Death > 7 days
Birth weight < 2,500 grams
Yes
No
NPIC
Chart
Both
740–759.9 = comprehensive range of all ICD-9-CM congenital
anomaly codes available _______ (write in diagnosis code)
778.0 = idiopathic fetal hydrops (not due to isoimmunization)
Congenital
anomalies
(Diagnosis
codes 740–
759.9)
Fetal hydrops
(Diagnosis code
778.0)
Death > 7 days
Birth weight <
2,500 grams
3. Uter ine ru pture
Diagnosis cod e 665.1 in
prima ry, rst or second
diagnosis code position only
Excludes D RG 665.1 in 3
diagnosi s code position
Yes
No
NPIC
Chart
Both
665.1 = rupture of uterus during labor
DRGs 370–375 (as above) _______ (write in diagnosis code)
Diagnosis code
665.1 in
diagnosis code
position
4. Materna l adm ission t o
ICU
DRG 370–375
ICU day or charge at any time
during stay (prior to or after
delivery)
Diagnosis code 640.xx–677.xx
with 5
th
digit = 2 (delivered with
mention of postpartum condition)
Excludes ICU stay or charges
without diagnosis code 640.xx–
677.xx and 5
th
digit = 2
Yes
No
NPIC
Chart
Both
640.xx–677.xx = includes complications and comorbidities
including hemorrhage, hypertension, infections of genitourinary
tract, other infections, anesthetic complications, other conditions in
mother complicating postpartum condition including diabetes,
thyroid dysfunction, anemia, drug dependence, mental disorders,
congenital cardiovascular disorders, etc.
DRGs 370–375 (as above) _______ (write in diagnosis code)
ICU stay or
charges without
diagnosis code
640.xx–677.xx
and 5
th
digit = 2
5. Birt h traum a
Inborn only (all gestational ages
included)
Diagnosis codes 767.0, 767.11,
767.3, 767.4, 767.5, 767.6,
767.7
Excludes outborn transferred in
Yes
No
NPIC
Chart
Both
767.0 = subdural and cerebral hemorrhage
767.11 = epicranial subaponeurotic hemorrhage - massive
767.3 = other injuries to the skeleton
767.4 = injury to spine and spinal column
767.5 = facial nerve injury
767.6 = injury to brachial plexus
767.7 = other cranial and peripheral nerve injury
Outborn
transferred in
6. Matern al Return to OR /
labor & delivery
DRG 370–375
Any of the following procedure
codes in rst or second position:
75.92, 69.02, 54 .61, 38.86,
39.98, 69.52
Excludes any other procedures
and inclusion list of procedure
codes but in 3 position
Yes
No
NPIC
Chart
Both
75.92 = evacuation of other hematoma of vulva or vagina ≤
69.02 = D&C following delivery
54.61 = reclosure of postoperative disruption of the abdominal wall
38.86 = other surgical occlusion of abdominal vessels
39.98 = control of hemorrhage
69.52 = aspiration curettage following delivery
DRG 370–375 (as above) _______ (write in diagnosis code)
Any other
procedures and
inclusion list of
procedure c odes
but in 3
rd
position
3
rd
rd
rd
Adverse Outcome Indicator:
(continued on page AP2)
Copyright 2015 The Joint Commission
The Joint Commission Journal on Quality and Patient Safety
Volume 41 Number 8August 2015
AP2
Online Only Content
Appendix 1. Data Collection Worksheet for Quality Outcomes Before and After TeamSTEPPS®
in Labor & Delivery (L&D) (continued)
Presen t Source Inclusion Cri teria (ICD -9-C M/DRG) Exclusion
Criteria
7. Admissio n to NICU
> 2,500 g & for > 24
hours †
Inborns only
2,500 grams
37 weeks gestation
ICU charge or LOS > 1 day
Within 1 day of birth
Excludes:
oCongenital anomalies: DRG
740–759.9
oFetal hydrops DRG 778.0
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
NPIC
Chart
Both
740–759.9 = comprehensive range of all ICD-9-CM congenital
anomaly codes available
778.0 = idiopathic fetal hydrop s (not due to isoimmunization
Congenital
anomalies:
diagnosis codes
740–759. 9
Fetal hydrops:
diagnosis code
778.0
8. APGA R < 7 at 5 minutes
Inborns only and
Birth weight
2,500 grams
Excludes congenital anomalies:
DRG 740–759.9
Excludes fetal hydrops DRG
778.0
NPIC
Chart
Both
740–759.9 = comprehensive range of all ICD-9-CM congenital
anomaly codes available
778.0 = idiopathic fetal hydrop s (not due to isoimmunization
Congenital
anomalies:
diagnosis codes
740–759.9
Fetal hydrops:
diagnosis code
778.0
9. Maternal B lood
transfu sion
Procedu re code 99.0 or Blood
Transfusion in dicator = 1
Excludes transfers into hospital
after delivery
NPIC
Chart
Both
99.0 = transfusion of blood and blood components
DRG 370–375 (as above) _______ (write in diagnosis code)
Transfers into
hospital after
delivery
10.3º- or 4 º-perineal tear
DRG 370–375
Diagnosis code 664.2 or 664.3
Excludes transfers into hospital
after delivery
NPIC
Chart
Both
664.2 = 3
rd
degree perineal laceration
664.3 = 4
th
degree perineal laceration
DRG 370–375 (as above) _______ (write in diagnosis code)
Transfers into
hospital after
delivery
Injury
Comments:
Chart r evie w adv erse
outcome consiste nt wi th
AOI a dmini strat ion d ata
algorithm i.e.
Is the iden tified Adve rse
Outco me co nfirmed p er
algorithm?
Is th e ICD9 code accurate
accor ding to definitio n?
Y
es
No
N/A
Yes
No
N/A
Yes
No
N/A
Comments:
Risk fa ctors /pre- existing
conditions relevan t to
adverse outco me
Y
es
No
N/A
Comments:
Chart r evie w adv erse
outcome consiste nt wi th
origi nally conc eived
adverse outco me i.e.
Is the Adver se Ou tcome
ident ified clin ically
relevan t and appropria te?
Y
es
No
N/A
Comments:
Other Comments (use back or attach sheet if needed):
Chart Reviewer’s Name:
Audit Date (MM/DD/YYYY)
Adverse Outcome Indicator:
Copyright 2015 The Joint Commission
... The AOI was broadened to include weighted scores for outcomes based on severity and named the Weighted Adverse Outcome Score (WAOS) ( Table 1). The AOI and WAOS have both been evaluated within an institutional setting as a quality improvement measure, but have not been broadly investigated [10][11][12]. The AOI has also been evaluated as a composite outcome during a randomized controlled trial [13]. ...
... We need novel methodologies to understand these well-established disparities so we can move towards studying intervention-based approaches. Tools like the AOI and WAOS have been applied in the context of institutional quality improvement measures but have not been broadly investigated [10][11][12]. Importantly, the scores of these tools were found to respond to patient safety interventions, showing real-world validity to the outcomes chosen in these tools [10,12,13]. There have been many calls to use implementation science and quality improvement tools to address perinatal disparities and one such methodology may be the application of standardized metrics such as the WAOS to quantify and track differences in outcomes [20]. ...
... The WAOS has demonstrated broad applicability: from identifying patients at risk of morbidity based on their medical history (i.e. preterm birth) to more macro-level usage during quality improvement review and comparison within and between institutions [10][11][12]23]. A modified version of the WAOS was applied to a tertiary care center before and after implementation of a labor and delivery intervention, and evaluation of outcomes demonstrated improvement in this modified measure after the care bundle was put into practice [12]. ...
Article
Full-text available
Objective To examine the association between the Weighted Adverse Outcome Score (WAOS) and race/ethnicity among a large and diverse population-based cohort of women and neonates in the United States. Study design This was a retrospective cohort study of women who delivered in the United States between 2011 and 2013. We identified mother-infant pairs with adverse maternal and/or neonatal outcomes. These outcomes were assigned weighted scores to account for relative severity. The association between race/ethnicity and WAOS was examined using chi-square test and multivariable logistic regression. Results Compared to White women and their neonates, Black women and their neonates were at higher odds of an adverse outcome. Conclusion(s) The vast majority of women and neonates had no adverse outcome. However, Black women and their neonates were found to have a higher WAOS. This tool could be used to designate hospitals or regions with higher-than-expected adverse outcomes and target them for intervention.
... The Composite Adverse Outcome Index (CAOI) [11][12][13], a composite evaluation of unfavorable events during the hospitalization time for delivery, was devised based on the occurrence of any of the following circumstances: (1) Maternal death occurring on hospital property; (2) newborn baby death within seven days of delivery with a birth weight less than 2500 g and gestational age < 37 weeks; (3) Ischemic respiratory hydrocephalus; (4) Uterine rupture; (5) Unanticipated Transfer of the Mother to the Intensive Care Unit; (6) Need for a Reoperation Within 72 Hours of Delivery; (7) bleeding after delivery needing a blood transfusion; (9) admittance of a baby within one day after childbirth to a neonatal intensive care unit or NICU, for a stay more than one day, (10) A fetal weight of less than 2500 g, an expected gestational age of approximately 37 weeks, and an Apgar score of less than 7 at five minutes, or (11) a neonatal birth injury. ...
... Globally, the panelists did not select any of the risk-adjusted indicators or adverse outcome index models suggested by various authors [12,[39][40][41]. However, they did select the main's cesarean indicator "nulliparous term singleton vertex cesarean birth" (NTSV CB) [42]. ...
Article
Full-text available
Background: Our research hypothesis was that most French indicators of quality of care have been validated by experts who are not clinicians and might not always be meaningful for clinicians. Our objective was to define a core set of measurable indicators of care quality during delivery and the immediate postpartum period relevant to clinical practice. Methods: A steering committee comprising nine specialists in obstetrics and/or public health conducted a literature review to develop potential indicators. A panel of obstetrician-gynecologists and midwives working in a delivery unit rated each indicator for appropriateness in a two-round Rand-modified Delphi procedure and a physical meeting. The consensus among the panelists was assessed. Results: In the first round, 145 panelists (110 obstetrician-gynecologists and 35 midwives) assessed 77 indicators and 3 definitions: 6 related to labor onset, 20 to delivery, 3 to pain management, 23 to neonatal morbidity/mortality, and 28 to maternal morbidity. In the second round, 132 panelists (98 obstetrician-gynecologists and 34 midwives) assessed 42 indicators and 1 definition. The final set comprised 50 indicators and 2 definitions. Conclusions: This Delphi procedure selected 50 indicators that reflect the quality of perinatal care. These indicators should be recorded in each French maternity ward's birth register for each delivery.
... This is a composite binomial outcome where presence of any of the included components confers a value of 1. The AOI is an obstetrical quality measure which has been validated as a measure of obstetrical patient safety in previous studies 21 . The index includes 10 adverse outcomes divided into maternal and fetal components. ...
Preprint
Full-text available
Objective: To discern the optimal plan for delivery in nulliparous women with obesity at term gestation. Subjects/Methods: This was a large population-based retrospective cohort study. It included nulliparous women with obesity (BMI>30) giving birth at a maternity hospital in Ontario, Canada with live, singleton, uncomplicated term gestations (37+0 to 41+6 weeks) between April 1st, 2012 and March 31st, 2019. A total of 27 472 deliveries were included. Interventions/Methods: Women were divided by plan for delivery (expectant management, induction of labour and no-labour caesarean section). The primary outcome was the Adverse Outcome Index (AOI), a binary composite of 10 maternal and neonatal adverse events. The Weighted Adverse Outcome Score (WAOS) was the secondary outcome. It provides a weighted score of each adverse event included in the AOI. Analyses were conducted using multivariable regression models. Analyses were stratified by each week of gestational age and by obesity class. Results No-labour caesarean section reduced the risk of adverse delivery outcome by 41% (aRR 0.59, 95%CI [0.50, 0.70]) compared to expectant management at term gestation. There was no statistically significant difference in adverse birth outcomes when comparing induction of labour to expectant management (aRR 1.03, 95% CI [0.96, 1.10]). The greatest benefit to no-labour caesarean section was observed in the reduction of adverse neonatal events (aRR 0.70, 95% CI [0.57, 0.87]) after 39 weeks of gestation. Conclusion In women with obesity, no-labour caesarean section reduces adverse birth outcomes.
... The primary outcome in this study was AOI. [12][13][14] The AOI was 4.9% within the Mayo Clinic ...
Article
Background : SARS-CoV-2 infection during pregnancy is associated with significant maternal morbidity and increased rates of preterm birth. For this reason, COVID-19 vaccine administration in pregnancy has been endorsed by multiple professional societies including ACOG and SMFM despite exclusion of pregnant women from initial clinical trials of vaccine safety and efficacy. However, to date little data exists regarding outcomes after COVID-19 vaccination of pregnant patients. Study Design : A comprehensive vaccine registry was combined with a delivery database for an integrated healthcare system to create a delivery cohort including vaccinated patients. Maternal sociodemographic data were examined to identify factors associated with COVID-19 vaccination. Pregnancy and birth outcomes were analyzed, including a composite measure of maternal and neonatal pregnancy complications, the Adverse Outcome Index. Results : Of 2002 patients in the delivery cohort, 140 (7.0%) received a COVID-19 vaccination during pregnancy and 212 (10.6%) experienced a COVID-19 infection during pregnancy. The median gestational age at first vaccination was 32 weeks (range 13 6/7-40 4/7), and patients vaccinated during pregnancy were less likely than unvaccinated patients to experience COVID-19 infection prior to delivery (1.4% (2/140) vs. 11.3% (210/1862), P<0.001). No maternal COVID-19 infections occurred after vaccination during pregnancy. Factors significantly associated with increased likelihood of vaccination in a multivariable logistic regression model included older age, higher level of maternal education, being a non-smoker, use of infertility treatment for the current pregnancy, and lower gravidity. No significant difference in the composite adverse outcome (5.0% (7/140) vs. 4.9% (91/1862), P=0.95) or other maternal or neonatal complications, including thromboembolic events and preterm birth, was observed in vaccinated compared to unvaccinated patients. Conclusions : Vaccinated pregnant women in this birth cohort were less likely to experience COVID-19 infection compared to unvaccinated pregnant patients, and COVID-19 vaccination during pregnancy was not associated with increased pregnancy or delivery complications. The cohort was skewed toward late pregnancy vaccination, and thus findings may not be generalizable to vaccination during early pregnancy.
... [40][41][42][43][44][45][46] Current metrics suffer from low reliability and validity scores, 47,48 for example, the Adverse Outcome Index should be modified to more appropriately measure preventable adverse events. 49 Moreover, health professionals, patients, and relatives should be involved in the design and collection of data 48,50,51 which should include patient-reported outcomes, morbidity, and cost, 52 Contracting strategies: Many healthcare systems use weighted capitation mechanisms for payment to general practitioners. In the ideal capitation model, several measures such as age, gender, morbidity, additional health needs, local labor costs, rurality, patient turnover, and so on can be included and comprehensively examined to predict patient expenditure and base capitation on the prediction. ...
Article
Full-text available
Background Increasing healthcare costs need to be contained in order to maintain equality of access to care for all EU citizens. A cross‐disciplinary consortium of experts was supported by the EU FP7 research programme, to produce a roadmap on cost containment, while maintaining or improving the quality of healthcare. The roadmap comprises two drivers: person‐centred care and health promotion; five critical enablers also need to be addressed: information technology, quality measures, infrastructure, incentive systems, and contracting strategies. Method In order to develop and test the roadmap, a COST Action project was initiated: COST−CARES, with 28 participating countries. This paper provides an overview of evidence about the effects of each of the identified enablers. Intersections between the drivers and the enablers are identified as critical for the success of future cost containment, in tandem with maintained or improved quality in healthcare. This will require further exploration through testing. Conclusion Cost containment of future healthcare, with maintained or improved quality, needs to be addressed through a concerted approach of testing key factors. We propose a framework for test lab design based on these drivers and enablers in different European countries.
... Dominant outcomes may influence the AOI and we caution readers against interpreting AOI findings independently of the WAOS and SI. Although limi tations exist, 49,50 they are validated measures that provide insight into patient safety. ...
Article
Full-text available
Background: Data on the effect of cesarean delivery on maternal request (CDMR) on maternal and neonatal outcomes are inconsistent and often limited by inadequate case definitions and other methodological issues. Our objective was to evaluate the trends, determinants and outcomes of CDMR using an intent-to-treat approach. Methods: We designed a population-based retrospective cohort study using data on low-risk pregnancies in Ontario, Canada (April 2012-March 2018). We assessed temporal trends and determinants of CDMR. We estimated the relative risks for component and composite outcomes used in the Adverse Outcome Index (AOI) related to planned CDMR compared with planned vaginal delivery using generalized estimating equation models. We compared the Weighted Adverse Outcome Score (WAOS) and the Severity Index (SI) across planned modes of delivery using analysis of variance. Results: Of 422 210 women, 0.4% (n = 1827) had a planned CDMR and 99.6% (n = 420 383) had a planned vaginal delivery. The prevalence of CDMR remained stable over time at 3.9% of all cesarean deliveries. Factors associated with CDMR included late maternal age, higher education, conception via in vitro fertilization, anxiety, nulliparity, being White, delivery at a hospital providing higher levels of maternal care and obstetrician-based antenatal care. Women who planned CDMR had a lower risk of adverse outcomes than women who planned vaginal delivery (adjusted relative risk 0.42, 95% confidence interval [CI] 0.33 to 0.53). The WAOS was lower for planned CDMR than planned vaginal delivery (mean difference -1.28, 95% CI -2.02 to -0.55). The SI was not statistically different between groups (mean difference 3.6, 95% CI -7.4 to 14.5). Interpretation: Rates of CDMR have not increased in Ontario. Planned CDMR is associated with a decreased risk of short-term adverse outcomes compared with planned vaginal delivery. Investigation into the long-term implications of CDMR is warranted.
... Also, the Adverse Outcome Index-5 (AOI-5) was calculated. The AOI-5 was designed to measure the magnitude of 5 adverse events that occurred during or around the delivery process [34]. It consists of perinatal mortality between a gestational age of 32 weeks and 7 days postpartum, neonatal intensive care unit (NICU) admission above 37 weeks, APGAR score lower than 7 after 5 min, postpartum haemorrhage and third-or fourth-degree perineal laceration. ...
Article
Full-text available
Background: With more than 20,000 asylum seekers arriving every year, healthcare for this population has become an important issue. Pregnant asylum seekers seem to be at risk of poor pregnancy outcomes. This study aimed to assess the difference in pregnancy outcomes between asylum seekers and the local Dutch population and to identify potential substandard factors of care. Methods: Using a retrospective study design we compared pregnancy outcomes of asylum-seeking and Dutch women who gave birth in a northern region of the Netherlands between January 2012 and December 2016. The following data were compared: perinatal mortality, maternal mortality, gestational age at delivery, preterm delivery, birth weight, small for gestational age children, APGAR score, intrauterine foetal death, mode of delivery and the need for pain medication. Cases of perinatal mortality in asylum seekers were reviewed for potential substandard factors. Results: A total of 344 Asylum-seeking women and 2323 Dutch women were included. Asylum seekers had a higher rate of perinatal mortality (3.2% vs. 0.6%, p = 0.000) including a higher rate of intrauterine foetal death (2.3% vs. 0.2%, p = 0.000), higher gestational age at birth (39 + 4 vs. 38 + 6 weeks, p = 0.000), labour was less often induced (36.9 vs. 43.8, p = 0.016), postnatal hospitalization was longer (2.24 vs. 1.72 days p = 0.006) and they received more opioid analgesics (27.3% vs. 22%, p = 0.029). Babies born from asylum-seeking women had lower birth weights (3265 vs. 3385 g, p = 0.000) and were more often small for gestational age (13.9% vs. 8.4%, p = 0.002). Multivariate analysis showed that the increased risk of perinatal mortality in asylum-seeking women was independent of parity, birth weight and gestational age at birth. Review of the perinatal mortality cases in asylum seekers revealed possible substandard factors, such as late initiation of antenatal care, missed appointments because of transportation problems, not recognising alarm symptoms, not knowing who to contact and transfer to other locations during pregnancy. Conclusion: Pregnant asylum seekers have an increased risk of adverse pregnancy outcomes. More research is needed to identify which specific risk factors are involved in poor perinatal outcomes in asylum seekers and to identify strategies to improve perinatal care for this group of vulnerable women.
Article
Background Patient safety is a key quality issue for health systems. Healthcare acquired adverse events (AEs) compromise safety and quality; therefore, their reporting and monitoring is a patient safety priority. Although administrative datasets are potentially efficient tools for monitoring rates of adverse events, concerns remain over the accuracy of their data. Chart review validation studies are required to explore the potential of administrative data to inform research and health policy. This review aims to present an overview of the methodological approaches and strategies used to validate rates of adverse events in administrative data through chart review. Methodology This review was conducted in line with the Joanna Briggs Institute methodological framework for scoping reviews. Through database searches, 1054 sources were identified, imported into Covidence, and screened against the inclusion criteria. Articles that validated rates of AEs in administrative data through chart review were included. Data were extracted, exported to Microsoft Excel, arranged into a charting table, and presented in a tabular and descriptive format. Results Fifty-six studies were included. Most sources reported on surgical adverse events however, other medical specialties were also explored. Chart reviews were used in all studies, however few agreed on terminology for the study design. Various methodological approaches and sampling strategies were used. Some studies used the Global Trigger Tool, a two-stage chart review method, whilst others used alternative single-, two-stage or unclear approaches. The sources used samples of flagged charts (n=24), flagged and random charts (n=11) and random charts (n=21). Most studies reported poor or moderate accuracy of adverse event rates. Some studies reported good accuracy of adverse event recording which highlights the potential of using administrative data for research purposes. Conclusions This review highlights the potential for administrative data to provide information on adverse event rates and improve patient safety and healthcare quality. Nonetheless, further work is warranted to ensure that administrative data are accurate. The variation of methodological approaches taken, and sampling techniques used demonstrate a lack of consensus on best practice, therefore, further clarity and consensus are necessary to develop a more systematic approach to chart reviewing.
Article
Full-text available
Purpose Obstetric adverse outcomes (AOs) are an important topic and the use of composite measures may favor the understanding of their impact on patient safety. The aim of the present study was to estimate AO frequency and obstetric care quality in low and high-risk maternity hospitals. Design/methodology/approach A one-year longitudinal follow-up study in two public Brazilian maternity hospitals. The frequency of AOs was measured in 2,880 randomly selected subjects, 1,440 in each institution, consisting of women and their newborn babies. The frequency of 14 AOs was estimated every two weeks for one year, as well as three obstetric care quality indices based on their frequency and severity as follows: the Adverse Outcome Index (AOI), the Weighted Adverse Outcome Score and the Severity Index. Findings A significant number of mothers and newborns exhibited AOs. The most prevalent maternal AOs were admission to the ICU and postpartum hysterectomy. Regarding newborns, hospitalization for > seven days and neonatal infection were the most common complications. Adverse outcomes were more frequent at the high-risk maternity, however, they were more severe at the low-risk facility. The AOI was stable at the high-risk center but declined after interventions during the follow-up year. Originality/value High AO frequency was identified in both mothers and newborns. The results demonstrate the need for public patient safety policies for low-risk maternity hospitals, where AOs were less frequent but more severe.
Article
Full-text available
No nationally accepted set of quality indicators exists in obstetrics. A set of 10 outcome measures and three quality improvement tools was developed as part of a study evaluating the effects of teamwork on obstetric care in 15 institutions and > 28,000 patients. Each outcome was assigned a severity weighting score. Three new obstetrical quality improvement outcome tools were developed. The Adverse Outcome Index (AOI) is the percent of deliveries with one or more adverse events. The average AOI during the pre-implementation data collection period of the teamwork study was 9.2% (range, 5.9%-16.6%). The Weighted Adverse Outcome Score (WAOS) describes the adverse event score per delivery. It is the sum of the points assigned to cases with adverse outcomes divided by the number of deliveries. The average WAOS for the preimplementation period was 3 points (range, 1.0-6.0). The Severity Index (SI) describes the severity of the outcomes. It is the sum of the adverse outcome scores divided by the number of deliveries with an identified adverse outcome. The average SI for the pre-implementation period was 31 points (range, 16-49). The outcome measures and the AOI, WAOS, and SI can be used to benchmark ongoing care within and among organizations. These tools may be useful nationally for determining quality obstetric care.
Article
We undertook a study to assess the reliability and validity of a new measure of low risk maternity care. A Perinatal Outcome Index (POI), which combines intrapartum process of care and clinical outcome items into a summary index score, was originally developed and evaluated in the Netherlands. It was designed to measure the extent to which a labour and birth are "optimal", that is, one with minimal intervention resulting in a healthy mother and a healthy baby. We modified the Dutch index to make it applicable to a Canadian setting. A panel of experts who were not connected with the study reviewed the modified version for applicability, feasibility of obtaining data easily, and content validity. Data were abstracted from the health records of 324 women in one hospital and two midwifery practices to obtain Perinatal Outcome Index scores and examine aspects of construct validity. We measured the inter-rater reliability of the research assistants who abstracted information. The panel achieved consensus on all items in the modified Perinatal Outcome Index to establish content (face) validity. Labour and birth data were readily obtained from health records with high inter-rater reliability (Kappa 0.78). In a linear regression model, birth at home, multiparity, and having a midwife or family physician as a care provider were significantly associated with higher scores (having a more optimal birth) and accounted for 37% of the score variance. The Perinatal Outcome Index has satisfied our expectations for content and construct validity. Research assistants found it easy to use and data items were readily available from women's health records. Inter-rater reliability was acceptable. We believe the modified index will be useful for comparative studies among women at low or average risk, and for quality assurance programs.
Article
Objective To estimate the incidence of multiple repeat caesarean section (MRCS) (five or more) in the UK and to describe the outcomes for women and their babies relative to women having fewer repeat caesarean sections. Design A national population-based prospective cohort study using the UK Obstetric Surveillance System (UKOSS). Setting All UK hospitals with consultant-led maternity units. Population Ninety-four women having their fifth or greater MRCS between January 2009 and December 2009, and 175 comparison women having their second to fourth caesarean section. Methods Prospective cohort and comparison identification through the UKOSS monthly mailing system. Main outcome measures Incidence, maternal and neonatal complications. Relative risk, unadjusted (OR) and adjusted (aOR) odds ratio estimates. Results The estimated UK incidence of MRCS was 1.20 per 10 000 maternities [95% confidence interval (CI), 0.97–1.47]. Women with MRCS had significantly more major obstetric haemorrhages (>1500 ml) (aOR, 18.6; 95% CI, 3.89–88.8), visceral damage (aOR, 17.6; 95% CI, 1.85–167.1) and critical care admissions (aOR, 15.5; 95% CI, 3.16–76.0), than women with lower order repeat caesarean sections. These risks were greatest in the 18% of women with MRCS who also had placenta praevia or accreta. Neonates of mothers having MRCS were significantly more likely to be born prior to 37 weeks of gestation (OR, 6.15; 95% CI, 2.56–15.78) and therefore had higher rates of complications and admissions. Conclusions MRCS is associated with greater maternal and neonatal morbidity than fewer caesarean sections. The associated maternal morbidity is largely secondary to placenta praevia and accreta, whereas higher rates of preterm delivery are most likely a response to antepartum haemorrhage.
Article
Concern regarding the association between cesarean delivery and long-term maternal morbidity is growing as the rate of cesarean delivery continues to increase. Observational evidence suggests that the risk of morbidity increases with increasing number of cesarean deliveries. The dominant maternal risk in subsequent pregnancies is placenta accreta spectrum disorder and its associated complications. A history of multiple cesarean deliveries is the major risk factor for this condition. Pregnancies following cesarean delivery also have increased risk for other types of abnormal placentation, reduced fetal growth, preterm birth, and possibly stillbirth. Chronic maternal morbidities associated with cesarean delivery include pelvic pain and adhesions. Adverse reproductive effects may include decreased fertility and increased risk of spontaneous abortion and ectopic pregnancy. Clinicians and patients need to be aware of the long-term risks associated with cesarean delivery so that they can be considered when determining the method of delivery for first and subsequent births.
Article
The purpose of this study was to assess the Adverse Outcome Index perinatal quality indicator system that was derived from administrative data. Adverse events were identified for 10 component measures; the Adverse Outcome Index was calculated by the National Perinatal Information Center from 42 months of administrative data. After retrospective chart review, we estimated positive predictive value for 10 measures that were obtained by corrected calculations of Adverse Outcome Index. Positive predictive values were 86-100% in 7 indicators, with lower values in 3 indicators: neonatal death, 0/2 fetuses; inborn birth trauma, 22/33 infants (67%); and maternal return to the operating room, 16/33 women (48.5%). In term admission to the neonatal intensive care unit, 107 false negatives were identified, with a negative predictive value of 45%. Indicator positive predictive value was variable. Performance can be strengthened by methods to identify both false-positive and false-negative adverse events that would include chart review and some measure specification revisions to improve alignment with original indicator intent. Interhospital comparison application requires further study.
Article
Evaluating perinatal outcomes within a framework of normalcy is a new focus of measurement. As maternal and child health clinicians and researchers look to evaluate care practices that are both of high quality and cost-effective, it is important to have measurement tools that assess differences among all women giving birth. The Optimality Index-US shifts the focus from rare adverse events to evidence-based optimal events. This article describes the continuing development of the index and discusses clinical implications for obstetric nurse clinicians.
Article
To evaluate the effect of teamwork training on the occurrence of adverse outcomes and process of care in labor and delivery. A cluster-randomized controlled trial was conducted at seven intervention and eight control hospitals. The intervention was a standardized teamwork training curriculum based on crew resource management that emphasized communication and team structure. The primary outcome was the proportion of deliveries at 20 weeks or more of gestation in which one or more adverse maternal or neonatal outcomes or both occurred (Adverse Outcome Index). Additional outcomes included 11 clinical process measures. A total of 1,307 personnel were trained and 28,536 deliveries analyzed. At baseline, there were no differences in demographic or delivery characteristics between the groups. The mean Adverse Outcome Index prevalence was similar in the control and intervention groups, both at baseline and after implementation of teamwork training (9.4% versus 9.0% and 7.2% versus 8.3%, respectively). The intracluster correlation coefficient was 0.015, with a resultant wide confidence interval for the difference in mean Adverse Outcome Index between groups (-5.6% to 3.2%). One process measure, the time from the decision to perform an immediate cesarean delivery to the incision, differed significantly after team training (33.3 minutes versus 21.2 minutes, P=.03). Training, as was conducted and implemented, did not transfer to a detectable impact in this study. The Adverse Outcome Index could be an important tool for comparing obstetric outcomes within and between institutions to help guide quality improvement. (www.ClinicalTrials.gov), NCT00381056 I.
Article
Unlabelled: Obstetric admissions are the leading cause of hospitalization for women in the United States, accounting for over 4 million hospital discharges each year. Measuring the quality of inpatient obstetrical care provided to these women is becoming increasingly important to patients, providers, and insurers. While numerous quality measures have been proposed, there is no agreement as to which measures should be used. An ideal quality measure for inpatient obstetrics would encompass 5 major characteristics: 1) association with meaningful maternal and neonatal outcomes, 2) relation to outcomes that are influenced by physician/health system behaviors, 3) affordability for application on a large scale basis, 4) acceptability to practicing obstetricians as a meaningful marker of quality, and 5) reliability/reproducibility. Traditional quality measurement tools such as maternal mortality, neonatal mortality and cesarean delivery rate are flawed measures. New measurements such as risk-adjusted primary cesarean rates, the nulliparous term singleton vertex cesarean birth (NTSV) rate, and the Adverse Outcomes Index (AOI) are currently being studied but these measures require further validation before widespread adoption. Target audience: Obstetricians & Gynecologists, Family Physicians Learning objectives: After completion of this article, the reader should be able to summarize that quality measures of inpatient obstetrical care are numerous, explain that no one agrees on which measures should be used, and state that newer measures, once validated, should be considered.
MD, formerly Chief of Clinical Operations, Western Regional Medical Command, and Obstetrics and Gynecology Consultant to US Army Surgeon General
  • Lisa M Foglia
Lisa M. Foglia, MD, is Residency Program Director, Department of Obstetrics and Gynecology, Madigan Army Medical Center, Tacoma, Washington. Peter E. Nielsen, MD, formerly Chief of Clinical Operations, Western Regional Medical Command, and Obstetrics and Gynecology Consultant to US Army Surgeon General, Joint Base Lewis-McChord, Washington, is Commander, General Leonard Wood Army Community Hospital, Fort Leonard Wood, Missouri.
EdD, is Perinatal Clinical Outcomes Research Nurse, Department of Obstetrics and Gynecology, Madigan Army Medical Center. Suzan Walker, MPH, is Perinatal Database Manager and Regional Perinatal Network Coordinator
  • Eileen A Hemann
Eileen A. Hemann, EdD, is Perinatal Clinical Outcomes Research Nurse, Department of Obstetrics and Gynecology, Madigan Army Medical Center. Suzan Walker, MPH, is Perinatal Database Manager and Regional Perinatal Network Coordinator, Pediatrics Division of Neonatology, University of Washington School of Medicine, Seattle.