Rare Adverse Medical Events in VA
Inpatient Care: Reliability Limits to
Using Patient Safety Indicators as
Alan N. West, William B. Weeks, and James P . Bagian
Objective. To assess Agency for Healthcare Research and Quality’s Patient Safety
Indicators (PSIs) as performance measures using Veterans Administration hospitaliza-
Data Sources Study Setting. Nine years (1997–2005) of all Veterans Health
Administration (VA) administrative hospital discharge data.
Study Design. Retrospective analysis using diagnoses and procedures to derive an-
nual rates and standard errors for 13 PSIs.
Data Collection/Extraction Methods. For either hospitals or hospital networks
(Veterans Integrated Service Networks [VISNs]), we calculated the percentages whose
for each PSI. We related our findings to the average annual number of adverse events
that each PSI represents. We also assessed time trends for the entire VA, by VISN, and
Principal Findings. PSI rates are more stablefor VISNs than for individualhospitals,
but only for those PSIs that reflect the most frequent adverse events. Only the most
frequent PSIs yield significant time trends, and only for larger systems.
Conclusions. Because they are so rare, PSIs are not reliable performance measures
to compare individual hospitals. The most frequent PSIs are more stable when applied
to hospital networks, but needing large patient samples nullifies their potential value to
managers seeking to improve quality locally or to patients seeking optimal care.
Key Words. Patient Safety, veterans, administrative data, reliability
Patient safety has become an issue of focal interest in the evaluation of health
care quality (Kohn, Corrigan, and Donaldson 1999). Recently the federal
Agency for Healthcare Research and Quality (AHRQ) has developed a com-
prehensive set of standardized patient safety measures, derived from hospital
discharge data, which potentially might be used to compare performance
rHealth Research and Educational Trust
within and between health care systems. In collaboration with the University
of California-Stanford Evidence-based Practice Center, AHRQ has put forth
data to calculate risk-adjusted rates of medical/surgical adverse events in hos-
pitalized patients (AHRQ 2003a; Zhan and Miller 2003a). Hospitalizations
involving adverse events identified by the PSIs have been associated with
worse outcomes and higher costs in both the private sector (Zhan and Miller
2003b) and the Veterans Health Administration (VA; Rosen et al. 2005).
AHRQ also has produced statistical software that calculates PSI rates and
variances from administrative hospital discharge data, using comorbidities to
risk-adjust them for cross-system comparisons. The intention is that the PSIs
eventually may help health care managers improve quality and medical con-
sumers choose higher quality services. Potentially, the PSIs may serve dual
purposes, including (a) case-finding to explore the local root causes of adverse
events for corrective intervention, and (b) establishing normative rates against
which asystem’s overallperformancecanbecompared.Animplication ofthe
second purpose, however, is that the reliabilities of the PSI rates must be
Studies of PSI rates in the VA (Rosen et al. 2005, 2006a, 2006b), the
nation’s largest health care system, have generated interest in their potential
application as performance measures. Because PSI rates are derived from
and objective measures of medical quality. But if they are to serve as adequate
performance measures they also must be both reliable and discriminating. As
some health care systems consistently perform better than others, if PSI rates
reflect underlying quality, they should be sufficiently sensitive to distinguish
systems from one another, and their rank orders should show some stability
over time. Rosen et al. (2006) showed that hospital-level PSI rates correlated
after a 3-year lag, but the proportions of variance accounted for were low,
ranging from 0.06 to 0.29; a shorter lag may have revealed better reliability.
Fortunately for health care consumers, the adverse medical events that
the PSIs identify are very uncommon occurrences. Unfortunately for health
Address correspondence to Alan N. West, Ph.D., VA Outcomes Group REAP, VA Medical
Center, White River Junction, VT 05009. William B. Weeks, M.D., MBA, is with the VA Out-
comes Group REAP, VA National Quality Scholars Fellowship Program, & Field Office of VA
National Center for Patient Safety, VA Medical Center, White River Junction, VT, and the
Departments of Psychiatry and Community & Family Medicine, Dartmouth Medical School,
Hanover, NH. James P. Bagian, M.D., PE, is with the VA National Center for Patient Safety, Ann
250HSR: Health Services Research 43:1, Part I (February 2008)
care analysts, the rareness of adverse events may limit the reliability of PSI
rates as performance indicators. PSIs vary widely in terms of how uncommon
the underlying events are. Some, such as Decubitus Ulcer, occur each year in
nearly every hospital, but others, such as Death in Low Mortality DRGs, or
algorithms do not even provide variance estimates for them from which risk-
adverse events, how well do they reflect changes in health care quality over
time or differences among providers? To how large a health care system
should they be applied to achieve reliable quality measures?
To address these questions we calculated annual PSI rates from 9 years
(1997–2005) of VA hospital discharge data. For each PSI, we assessed stabil-
regional Veterans Integrated Service Networks (VISNs) into which the VA
system is organized (in 2002, two VISNs were merged into one, but we kept
their data separate for these analyses), and for each VA Medical Center be-
longing to these VISNs. We anticipated that from year to year, PSIs would be
more stable if they are calculated for larger health care systems, i.e., VISNs,
than at the hospital level. We expected, as well, that those PSIs that are based
on higher numbers of underlying adverse events would be more stable over
We applied AHRQ’s algorithms for finding adverse events and calculating
PSI rates to all VA hospital discharge data for federal fiscal years 1997–2005
inclusive. VA’s administrative database for hospitalizations is its annual
which includes demographics as well as admission and discharge dates, and
ICD-9-CM diagnoses and procedures, for each hospitalization and any
specialty service portion thereof.
PTF data include both acute and nonacute (e.g., long-term care) admis-
sions, and As the PSIs were developed for acute admissions only, we elim-
inated all nonacute admissions as well as the nonacute portions of mixed
admissions, following the procedures developed by Rosen, Rivard, and their
colleagues (Rivard et al. 2005; Rosen et al. 2005). The method involves iden-
tifying each bedsection (admission to a particular specialty service during the
hospitalization) in a mixed admission as either acute or nonacutebased on the
Rare Adverse Medical Events in VA Inpatient Care251
definitions of VA’s Health Economics Resource Center, and then splitting
create new hospitalization records. Any new record may have a new admis-
sion or discharge date, a new principal diagnosis derived from its first bed-
section, or a new DRG defined as the highest cost-weighted DRG for the new
admission. The method also entails an algorithm for identifying the principal
procedure for an admission (not specified in PTF data) by searching for valid
operating room procedures (based on a DRG grouping list) and selecting the
chronologically first procedure in either the Procedure or Surgery file. Ex-
ploiting the richness of the PTF data, the method eliminates nonelective hos-
pitalizations by screening out nonelective DRG codes, surgeries that were
performed on the third day of the admission or later, and procedures done at
timesotherthanweekdaysbetween5 A.M.and5 P.M.Fromtheresultingdataset
for each year, we then derived the input variables to submit to the PSI al-
gorithms, following AHRQ’s specifications (AHRQ 2003b); these variables
include patient demographics, date of admission, DRG, diagnoses, proce-
dures and their dates, and VISN or hospital code.
AHRQ has made available on its website (www.ahrq.gov) a set of SAS
programs to be applied to prepared data to compute its PSIs; we used version
3.0, which was released in February 2006. The programs implement algo-
rithms that find patients at risk for each PSI as well as those who actually
experience the adverse event. Observed rates are calculated for all PSIs, and
most PSIs are then risk-adjusted (using comorbidity weights derived from
nationally normative hospitalization data) and given confidence intervals.
Some PSIs do not yield risk-adjusted rates or confidence intervals because
to obstetrical DRGs, which are extremely rare among VA’s predominantly
all yield risk-adjusted rates:
Complications of Anesthesia (1)
Decubitus Ulcer (3)
Failure to Rescue (4)
Iatrogenic Pneumothorax (6)
Selected Infections Due to Medical Care (7)
Postoperative Hip Fracture (8)
Postoperative Hemorrhage or Hematoma (9)
Postoperative Physiologic & Metabolic Derangement (10)
Postoperative Respiratory Failure (11)
252HSR: Health Services Research 43:1, Part I (February 2008)
Postoperative Pulmonary Embolism or Deep Vein Thrombosis (12)
Postoperative Sepsis (13)
Postoperative Wound Dehiscence (14)
Accidental Puncture or Laceration (15)
The specific diagnoses and procedures defining these PSIs have been
well described elsewhere (AHRQ 2003a; Rosen et al. 2005); it suffices here to
notethatthese13 PSIsassessa widerange ofmedical/surgicalproceduresand
From the risk-adjusted PSI rates AHRQ’s SAS programs also calculate
smoothed rates with standard errors, using multivariate signal extraction
(AHRQ 2003b, pp. 38–39). Smoothing removes ‘‘noise’’ in the risk-adjusted
rates, which may be greater for smaller health care systems or very rare ad-
verse events, by adjusting rates toward the overall mean. For this purpose,
AHRQ calculated ‘‘shrinkage’’ estimates from HCUP Year 2002 SID data
only, as the smoothed rates reduce outlying values due to smaller Ns and
therefore should yield better reliability, but we note that our findings were
similar when we submitted the intermediate risk-adjusted rates to the same
hospitals that had any patients at risk for the given PSI. Findings were related
to the annual frequencies at which the underlying adverse medical events
occur. Averaging the annual smoothed PSI rates and standard errors pro-
duced by the AHRQ software across the 9 years, 1997–2005, we calculated
overall rate/SE ratios, whose magnitudes reflect the power of a PSI to yield
reliable differences in statistical comparisons (rather like the inverse of a co-
rates. Additionally, for each PSI we identified those VISNs or hospitals whose
rates were among the highest, middle, or lowest third in 2001, the central year
in our range. Using 2001 as our reference, we then calculated the percentages
of VISNs or hospitals that also were among the highest, middle, or lowest
thirds in other years (a measure we call ‘‘retention’’ whether before or after) to
determine whether indicators or health care systems of different size differ
with respect to stability over time. As stability should also help general trends
emerge, we considered how well PSIs of different size reveal time trends for
the VA as a whole and within VISNs or hospitals.
Rare Adverse Medical Events in VA Inpatient Care253
We applied the AHRQ programs to calculate, for each PSI and for each year,
them, (c) the risk-adjusted, smoothed PSI rate, and (d) the standard error for
thisrate. These calculations were performedat eachof three levels of analysis:
(1) for the VA system nationwide, (2) at the VISN level, and (3) at the hospital
level. Within each year, VISN-level data were averaged across the 22 VISNs;
similarly, each year’s hospital-level data were averaged across those hospitals
the PSI. For each PSI and level of analysis, we then averaged across the 9
the resulting annual averages (PSI rates and standard errors are expressed as
cases per 10,000 patients at risk). The ratio of average annual PSI rate to
average standard error is given also, as this measure reflects the potential
strength of the PSI to detect differences statistically, much like the inverse of a
coefficient of variation. The PSIs are listed in order of frequency of adverse
VA-wide there are a few thousand Failure to Rescue or Decubitus Ulcer
events each year. Accidental Punctures and Postoperative Deep Vein Throm-
bosis each account for more than a thousand events annually, as well. The
other PSIs are considerably less frequent. Failure to Rescue also is much more
likely to occur among patients at risk for it, with nearly 15 percent suffering
death. Many more patients are at risk for Decubitus Ulcer, but fewer than 2
percentof them experience it.Other PSIs are muchlower risks. At the level of
at either the VISN or hospital level, correlations between different PSIs are
low, rarely exceeding r50.40 (principal components analyses yield several
components, each accounting for little variance), which suggests that the PSIs
vary independently of one another.
Table 1 also shows that the average standard errors of per hospital PSI
rates are considerably greater than those for per VISN rates, which in turn are
likelihood that comparisons among hospitals will yield statistically reliable
differences, whereas comparisons among VISNs or repeated measures on
national data might. The ratio of average PSI rate to average standard error,
which reflects the potential statistical strength of comparisons, drops consid-
of PSI rate to standard error for individual VISNs or hospitals in each year
254HSR: Health Services Research 43:1, Part I (February 2008)
Mean Annual Adverse Medical Events, Cases at Risk, Smoothed Patient Safety Indicators (PSI) Rates, and
Standard Errors for 1997–2005
For the entire VAn
Rare Adverse Medical Events in VA Inpatient Care255
Note: Mean PSI rates and standard errors are expressed as cases per 10,000 patients at risk.
nAveraged across all 9 years, 1997–2005.
wAveraged across all 22 VISNs within each year, then across all 9 years.
zAveraged across all hospitals within each year, then across all 9 years.
256HSR: Health Services Research 43:1, Part I (February 2008)
(data not shown in the table), there were numerous instances, particularly for
the less frequent PSIs, where the ratio was o2; as the ratio is comparable to a
from zero. Consequently, comparisons among VISNs or time series analyses
of national data using the most frequent PSIs are more likely to yield reliable
differences than comparisons among hospitals on the less common PSIs.
To illustrate this point, we tested the annual PSI rates for the entire VA
system nationwide forlinearor higher degree trends from 1997 through 2005.
Only four PSIs yielded statistically significant time trends, which we show
(with standard errors) in Figure 1. All four PSIs represent strongly significant
linear trends (tested via a z score calculated as linear slope divided by the ratio
of mean annual standard error to the standard deviation of number of years).
For the most frequent PSI, Failure to Rescue, the smoothed rate dropped with
time, z5 ?8.04, po.0001. But forthe otherthree,there were strongincreases
with time: Decubitus Ulcer, z53.29, po.001; Postoperative DVT, z53.37,
po.001; and Accidental Puncture or Laceration, z57.14, po.0001. For Fail-
ure to Rescue, there was a sharp drop in numbers of adverse events (20
percent) from 1997 to 1999, and then steady annual counts in the context of
increasing numbers of patients at risk (data not shown). For the latter three
PSIs, annual numbers of adverse events rose linearly with time, while patients
at risk remained fairly constant. The strong linearity of these trends suggests
years that may relate to care provision or alternatively to different coding
practices. However, these processes have not been reflected in the rates of the
other, less common, PSIs. Though their greater variability made the linear
and Postoperative Sepsis, also rose more than 20 percent during this time; for
each, patients at risk rose across years, but adverse events per year rose faster.
Rates for the other PSIs show no clear time trends.
When we looked fortime effects withinindividual VISNs, onlythemost
frequent PSIs yielded significant trends: At po.05 or better, six (of 22) VISNs
having linear trends for Decubitus Ulcer and Selected Infections Due to
Medical Care, and another joining five other VISNs in showing linear trends
for Failure to Rescue; several other VISNs trended nonsignificantly in the
same directions on these PSIs. On the other hand, significant time trends for
individual hospitals were slightly fewer than would be expected by chance. In
short, time trends emerged only for the more frequent PSIs, at either the
national or VISN level, but not at the individual hospital level above chance.
Rare Adverse Medical Events in VA Inpatient Care 257
great relative to rates to yield significant trends.
One way to assess a PSI’s reliability is to consider how well one year’s
rate predicts rates in other years. For each PSI we calculated an intraclass
Smoothed Rate for Failure to Rescue
Rate for Decubitus Ulcer, Post-Op DVT, Acc'l Puncture
Failure to RescueDecubitus Ulcer
Post-Op DVT Accidental Puncture
Health Care System, Which Are the Only PSIs to Yield Significant Time
Trends at the National Level.
Annual Smoothed Rates (and Standard Errors), for the Four Most
Note: Left y-axis is smoothed rate for Failure to Rescue; right y-axis is for Decubitus
Ulcer, Post-Op DVT, and Accidental Puncture.
258HSR: Health Services Research 43:1, Part I (February 2008)
correlation from a general linear models analysis of PSI rates between 1997
and 2005; we calculated each PSI’s intraclass correlation twice, once using
VISNs and once using hospitals as the units of analysis. Figure 2 shows these
events throughout the VA. Generally, reliability over time is better for those
PSIs that are based on the most adverse events, although for Failure to Rescue
reliability is low despite its comparatively large number of underlying events.
For only three PSIs, Decubitus Ulcer, Postoperative Respiratory Failure, and
Postoperative Pulmonary Embolism or Deep Vein Thrombosis, did annual
rates predict more than 50 percent of the variance in other years. Hospital-
level correlations (not shown) are consistently lower across allPSIs, so that the
than 30 percent. On other PSIs, VISNs and hospitals changed their rank
orders considerably from year to year.
8 Postop Hip Fracture
13 Postop Sepsis
1 Comps Anesthesia
10 Postop Phys/Metab
14 Postop Wound
9 Postop Hemorrhage
6 Iatro Pneumothorax
11 Postop Resp Fail
7 Infections Due to Care
12 Postop DVT
4 Decub Ulcer
3 Fail Rescue
15 Acc'l Puncture
0 5001,0001,5002,0002,500 3,000 3,500
(PSI) Rates, for VISNs across 9 Years, 1997–2005, Plotted against Average
Annual Number (throughout the VA) of PSI Adverse Events (Rate
Explains 450 percent of Variance in Other Years’ Rates for PSIs 4, 11,
and 12 Only).
IntraClass Correlations of Smoothed Patient Safety Indicator
Rare Adverse Medical Events in VA Inpatient Care 259
There were, however, several VISNs, as well as hospitals, whose PSI
PSI, we rank-ordered the VISNs on their smoothed rates for 2001, the central
year in our 9-year time span, and apportioned them to the highest, middle, or
8 years, 1997–2000 and 2002–2005, to determine whether each VISN re-
mained in the same third; we calculated the percentage that did so in each
year, under the assumption that if this ‘‘retention’’ rate is higher, the PSI is
more stable over time. We repeated these calculations using hospitals as the
and post-2001 years differ. Figure 3 shows these percentages averaged across
the 8 years (VISN-level data: upper plot; hospital-level: lower).
The figure suggests that for VISN-level data retention in the same third
the less frequent PSIs. To test this notion, we submitted the annual retention
and 2002–2005, as units of analysis, and including PSI and 2001 ranking
(highest, middle, or lowest third) as crossed factors. Retention rate varied
significantly across PSIs, F(12,311)518.94, po.0001. Retention also was
greater overall for VISNs with the lowest third of PSI rates (50 percent) or the
highest rates (51 percent) than for VISNs in the middle third (43 percent),
F(24,311)51.46, po.08, as differences among thirds occurred primarily for
themore frequent PSIs. We confirmedthisin afollow-up analysisinwhich we
combined the three most frequent PSIs, Failure to Rescue, Decubitus Ulcer,
finding a significant interaction, F(2,311)54.43, po.05. VISNs whose rates
are among the highestor lowest thirds on the most common PSIs are the most
likely to remain at the same levels over time, whereas stability across years on
the most infrequent PSIs is much closer to chance.
The hospital-level data in Figure 3 show much different patterns, with
less retention overall than for VISNs, though again average retention varied
significantly across PSIs, F(12,311)523.43, po.0001. Quality thirds also
differed in their average retention rates, F(2,311)59.03, po.001, but the
differences were not great (lowest rates: 52 percent, middle: 49 percent,
highest rates: 50 percent). For hospitals, however, there was a significant
interaction, F(24,311)511.07, po.0001, due to an anomalous finding that
260HSR: Health Services Research 43:1, Part I (February 2008)
middle-level hospitals had higher retention rates when quality rankings were
PSIs to all others, there also was a significant interaction, F(2,311)521.08,
Middle PSI Rates
43 15 127 1169 14131018
Lowest PSI RatesHighest PSI Rates
Middle PSI Rates
43 15 127 1169 14 131018
Lowest PSI RatesHighest PSI Rates
the Lowest, Middle, or Highest Third of Smoothed Patient Safety Indicator
(PSI) Rates in 2001, and Also Were in the Same Third in Other Years.
Percentages in the Same Third in Each of the Other 8 Years Were Averaged.
PSIs Are Listed in Order of Frequency.
Average Percentages of VISNs, or Hospitals, That Were, among
Rare Adverse Medical Events in VA Inpatient Care261
po.0001, revealing that only the most frequent PSIs yielded higher retention
rates for hospitals in the highest or lowest thirds of PSI rate rankings. In
summary, rankings based on the more common PSIs are more stable, par-
ticularly for larger health care systems, and for extreme rather than middle
rankings. Most PSIs are too unstable to yield reliable comparisons among
health care systems, particularly at the hospital level.
AHRQ has proposed multiple PSIs to broadly assess medical/surgical quality
in inpatient settings. But as AHRQ acknowledges (AHRQ 2003a), the differ-
ent PSIs cannot be given equal weight as the annual numbers of adverse
though relatively few patients are at risk for it, among these, 15 percent die.
The VA can congratulate itself that during these recent 9 years the smoothed
rate for this most common PSI has dropped by more than 25 percent nation-
ally. The next most common PSIs, Decubitus Ulcer, Accidental Puncture or
Laceration, and Postoperative Pulmonary Embolism or Deep Vein Throm-
bosis, also represent thousands of adverse events per year, and for these there
have been steady increases in reported incidence over the same time, which
may indicate a need for case-finding and focused interventions to improve
quality, changes in coding practices during a time period when VA markedly
increased commercial insurance billings, or simply a trend of increased re-
porting of adverse events. Other PSIs reflect many fewer adverse events an-
time trends do not emerge. PSIs vary independently of one another, within
and across years, so that global conclusions about patient safety within VISNs
or hospitals are not possible.
underlying them, we sought to assess their stability as health care quality
measures in systems of different size, i.e., the VA as a whole; the health care
networks, VISNs, into which the VA has been organized; and the individual
hospitals. We found that intraclass correlations of PSI rates are substantially
PSIs; uncommon PSIs had relatively low correlations, regardless of the size of
the system. Similarly, when we considered how well PSI-defined groups of
VISNs or hospitals remained stable over time, we found that rankings based
262HSR: Health Services Research 43:1, Part I (February 2008)
on the more common PSIs are more stable for the VISNs, and for extreme
rather than middle rankings, and are considerably less stable at the hospital
level and for the less common PSIs. Our findings suggest that a health care
system (hospital or network) cannot expect a given PSI rate to be a reliable
performance indicator unless its patients at risk number in the thousands and
adverse events number in the hundreds. Consequently, only the largest sys-
tems might reasonably consider using PSI rates to assess quality of care. But
theneedfor such large patientsamples must necessarily limit the usefulness of
PSI rates to managers seeking to improve health care quality or patients
seeking optimal care.
This study is limited, in part, because AHRQ PSIs are calculated from
administrative data. To the degree that important data elements are not
recorded in discharge summaries, or there are variations in coding across
providers within the VA, our analyses and conclusions may be inaccurate.
Another limitation may stem from our assumption that efforts to reduce
adverse events measured by the PSIs likely have been randomly dispersed
in time among VISNs or hospitals during the time period examined. This is
not to say that the VA has not made a concerted effort to improve
patient safety during this time period; it has (Weeks and Bagian 2000; Bagian
et al. 2001, 2002; Best et al. 2002; Heget et al. 2002; Neily et al. 2003;
Mills et al. 2004, 2006; Eldridge et al. 2006). We simply have assumed that
these efforts were not systematically distributed in time across VA hospitals
Most importantly, AHRQ PSIs have not yet been strongly validated
within the VA or other health care systems. At the least, a large-scale study
reviewing clinical recordswillbeneeded to assess howwellthePSIs represent
the documented incidence of preventable adverse events. Although prior
studies (Zhan and Miller 2003b; Rosen et al. 2005) found higher mortality,
lengths of stay, and costs associated with admissions involving PSI events,
there is some evidence that the sensitivity/specificity of AHRQ PSIs is less
data collected through VA’s National Surgical Quality Improvement Project
(Best et al. 2002) found the PSIs had sensitivities of between 37 and 67 percent
system (Bagian et al. 2002) and found that only 47 percent were detected by
the AHRQ PSI software. Additionally, an AHRQ researcher (A. Elixhauser,
personal communication) told us of recent evidence that roughly eight of
every nine Decubitus Ulcer cases may have been present on admission to the
Rare Adverse Medical Events in VA Inpatient Care263
hospital, suggesting a need for additional data elements to detect true adverse
Because they do not reflect health care system performance consistently
over time, it is quite premature to use PSI rates as performance measures; for
now, their use should be limited to research that will test their validity. If they
are presented as reliable performance measures before validation, hospital or
respond rapidly to correct them, which may be more detrimental to than
supportive of improved operations. Given the rareness of these adverse
events, a much better approach is to use root cause analysis to understand and
address system vulnerabilities, perhaps using adverse events identified by the
PSI algorithms and validated through chart review as the triggering mech-
anism for focused inquiry. Consistency in rates for the most common PSIs
may potentiallyserve toidentify thosehospital networksthatareexemplars of
superlative care, or those that may require special attention for improvement
efforts. But at this time, PSI rate comparisons should not provide the basis
for guiding managers in local quality improvement efforts or health care
consumers in selecting high quality providers.
This research was funded by two Veterans Health Administration grants: The
Research Enhancement Award Program (REAP; grant # REA 03-098) and a
grant from VHA’s Health Services Research & Development Service (grant #
Disclosures: The authors have no conflicts of interest relating to this
Disclaimers: The views expressed in this article do not necessarily rep-
resent the views of the Department of Veterans Affairs or of the United States
Agency for Healthcare Research and Quality (AHRQ). 2003a. AHRQ Quality Indica-
tors–Guide to Patient Safety Indicators. Version 2.1, Revision 2. Rockville, MD:
October 22, 2004. AHRQ Pub.03-R203.
264HSR: Health Services Research 43:1, Part I (February 2008)
—— —— ——. 2003b. AHRQ Quality Indicators–Patient Safety Indicators: Software Documentation.
Version 2.1–SAS, Revision 3a. Rockville, MD: February 15, 2005. AHRQ Pub.03-
Bagian, J. P., J. Gosbee, C. Z. Lee, L. Williams, S. D. McKnight, and D. M. Mannos.
2002. ‘‘The Veterans Affairs Root Cause Analysis System in Action.’’ Joint Com-
mission Journal on Quality Improvement 28 (10): 531–45.
Bagian, J. P., C. Lee, J. Gosbee, J. DeRosier, E. Stalhandske, N. Eldridge, R. Williams,
and M. Burkhardt. 2001. ‘‘Developing and Deploying a Patient Safety Program
in a Large Health Care Delivery System: You Can’t Fix What You Don’t Know
About.’’ Joint Commission Journal on Quality Improvement 27 (10): 522–32.
Best, W. R., S. F. Khuri, M. Phelan, K. Hur, W. G. Henderson, J. G. Demakis, and J.
Daley. 2002. ‘‘Identifying Patient Preoperative Risk Factors and Postoperative
Adverse Events in Administrative Databases: Results from the Department of
American College of Surgeons 194: 257–66.
Eldridge, N. E.,S. S. Woods,R. S. Bonello, K. Clutter,L. Ellingson,M. A. Harris, B.K.
Reichling, G. A. Roselle, and S. M. Wright. 2006. ‘‘Using the Six Sigma Process
to Implement the Centers for Disease Control and Prevention Guideline for
Hand Hygiene in 4 Intensive Care Units.’’ Journal of General Internal Medicine 21
(Suppl. 2): S35–42.
Safety Awards. System Innovation: Veterans Health Administration National
Center for Patient Safety.’’ Joint Commission Journal on Quality Improvement 289
Kohn, L., J. Corrigan, M. Donaldson. 1999. To Err Is Human: Building a Safer Health
System, editedby the Instituteof Medicine. Washington, DC:NationalAcademy
Mills, P. D., J. M. DeRosier, J. Neily, S. D. McKnight, W. B. Weeks, and J. P. Bagian.
about It.’’ Joint Commission Journal on Quality and Safety 30 (9): 488–96.
Mills, P.D., J.Neily,E. Mims, M. E.Burkhardt, andJ.P. Bagian. 2006. ‘‘Improvingthe
Bar-Coded Medication Administration System at the Department of Veterans
Affairs.’’ American Journal of Health-System Pharmacy 63 (15): 1442–7.
Neily, J., G. Ogrinc, P. Mills, R. Williams, E. Stalhandske, J. Bagian, and W. B. Weeks.
2003. ‘‘Using Aggregate Root Cause Analysis to Improve Patient Safety.’’ Joint
Commission Journal on Quality and Safety 29 (8): 434–9.
Rivard, P. E., A. R. Elwy, S. Loveland, S. Zhao, D. Tsilimingras, A. Elixhauser, P. S.
Romano, and A. K. Rosen. 2005. ‘‘Applying Patient Safety Indicators (PSIs)
across Healthcare Systems: Achieving Data Comparability.’’ In Advances in
Patient Safety: From Research to Implementation, edited by K. Henriksen, J. B. Bat-
tles,E.Marks,andD.I.Lewin,pp. 7–25. Rockville, MD:AgencyforHealthcare
Research and Quality.
Rosen, A. K., P. Rivard, S. Zhao, S. Loveland, D. Tsilimingras, C. L. Christiansen, A.
Elixhauser, and P. S. Romano. 2005. ‘‘Evaluating the Patient Safety Indicators:
Rare Adverse Medical Events in VA Inpatient Care 265
HowWell Do They Perform on Veterans HealthAdministration Data?’’Medical Download full-text
Care 43 (9): 873–84.
Rosen, A. K., P. Rivard, S. Zhao, D. Tsilimingas, S. Loveland, C. Christiansen, W.
Henderson, S.F.Khuri, A.Elixhauser, andP.Romano. 2006a. ‘‘Identification of
Annual HSR&D Service Meeting, Washington, DC.
Rosen, A. K., S. Zhao, P. Rivard, S. Loveland, M. E. Montez-Rath, A. Elixhauser, and
P. S. Romano. 2006b. ‘‘Tracking Rates of Patient Safety Indicators over Time:
Lessons from the Veterans Administration.’’ Medical Care 44 (9): 850–61.
Weeks, W. B., and J. P. Bagian. 2000. ‘‘Developing a Culture of Safety in the Veterans
Health Administration.’’ Effective Clinical Practice 3: 270–6.
Zhan, C., and M. Miller. 2003a. ‘‘Administrative Data Based Patient Safety Research:
A Critical Review.’’ Quality and Safety in Health Care 12 (Suppl. 2): 58ii–63ii.
—— —— ——. 2003b. ‘‘Excess Length of Stay, Charges, and MortalityAttributable to Medical
Injuries during Hospitalization.’’ Journal of the American Medical Association 290
266HSR: Health Services Research 43:1, Part I (February 2008)