Validity of Selected AHRQ Patient Safety Indicators Based on VA National Surgical Quality Improvement Program Data

UC Davis Division of General Medicine and Center for Healthcare Policy and Research, 4150 V Street, PSSB Suite 2400, Sacramento, CA 95817, USA.
Health Services Research (Impact Factor: 2.78). 10/2008; 44(1):182-204. DOI: 10.1111/j.1475-6773.2008.00905.x
Source: PubMed


To examine the criterion validity of the Agency for Health Care Research and Quality (AHRQ) Patient Safety Indicators (PSIs) using clinical data from the Veterans Health Administration (VA) National Surgical Quality Improvement Program (NSQIP).
Fifty five thousand seven hundred and fifty two matched hospitalizations from 2001 VA inpatient surgical discharge data and NSQIP chart-abstracted data.
We examined the sensitivities, specificities, positive predictive values (PPVs), and positive likelihood ratios of five surgical PSIs that corresponded to NSQIP adverse events. We created and tested alternative definitions of each PSI.
FY01 inpatient discharge data were merged with 2001 NSQIP data abstracted from medical records for major noncardiac surgeries.
Sensitivities were 19-56 percent for original PSI definitions; and 37-63 percent using alternative PSI definitions. PPVs were 22-74 percent and did not improve with modifications. Positive likelihood ratios were 65-524 using original definitions, and 64-744 using alternative definitions. "Postoperative respiratory failure" and "postoperative wound dehiscence" exhibited significant increases in sensitivity after modifications.
PSI sensitivities and PPVs were moderate. For three of the five PSIs, AHRQ has incorporated our alternative, higher sensitivity definitions into current PSI algorithms. Further validation should be considered before most of the PSIs evaluated herein are used to publicly compare or reward hospital performance.

Download full-text


Available from: Patrick S Romano
  • Source
    • "g fusion surgery has not been rigorously validated through a comparison of chart reviews . However , readmission , infections , mortality , and life - threaten - ing complications are part of the National Surgical Quality Improvement Program ( NSQIP ) , which has been used as the gold standard to improve the validity of patient safety indicators ( Romano et al . 2009 ) . Furthermore , our estimates for repeat surgery and mortality following fusion surgery are similar to those reported in both administrative and clinical studies ( Malter et al . 1998 ; Martin et al . 2007 , 2011 ; Juratli et al . 2009 ) . Future research efforts should focus on the validation and improvement of claims - based methods"
    [Show abstract] [Hide abstract]
    ABSTRACT: To identify factors that account for variation in complication rates across hospitals and surgeons performing lumbar spinal fusion surgery. Discharge registry including all nonfederal hospitals in Washington State from 2004 to 2007. We identified adults (n = 6,091) undergoing an initial inpatient lumbar fusion for degenerative conditions. We identified whether each patient had a subsequent complication within 90 days. Logistic regression models with hospital and surgeon random effects were used to examine complications, controlling for patient characteristics and comorbidity. Complications within 90 days of a fusion occurred in 4.8 percent of patients, and 2.2 percent had a reoperation. Hospital effects accounted for 8.8 percent of the total variability, and surgeon effects account for 14.4 percent. Surgeon factors account for 54.5 percent of the variation in hospital reoperation rates, and 47.2 percent of the variation in hospital complication rates. The discretionary use of operative features, such as the inclusion of bone morphogenetic proteins, accounted for 30 and 50 percent of the variation in surgeons' reoperation and complication rates, respectively. To improve the safety of lumbar spinal fusion surgery, quality improvement efforts that focus on surgeons' discretionary use of operative techniques may be more effective than those that target hospitals.
    Full-text · Article · Jun 2012 · Health Services Research
  • Source
    • "In spite of the efforts made in building a valid tool concerns remain about whether PSI are appropriate to inform hospital performance. Beyond the need of local adaptation [12,13], most of the caveats have pointed out to flaws in their capacity to attribute excess-cases to hospitals by detecting true incident adverse events [14-23]. Less has been written on their empirical properties, mainly because of their local nature; in particular, to what extent PSI show systematic variation on adjusted-incidence (as opposed to random) and, their ability to provide precise estimates and therefore, being sensitive to detect providers over the expected. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Patient Safety Indicators (PSI) are being modestly used in Spain, somewhat due to concerns on their empirical properties. This paper provides evidence by answering three questions: a) Are PSI differences across hospitals systematic -rather than random?; b) Do PSI measure differences among hospital-providers -as opposed to differences among patients?; and, c) Are measurements able to detect hospitals with a higher than "expected" number of cases? An empirical validation study on administrative data was carried out. All 2005 and 2006 publicly-funded hospital discharges were used to retrieve eligible cases of five PSI: Death in low-mortality DRGs (MLM); decubitus ulcer (DU); postoperative pulmonary embolism or deep-vein thrombosis (PE-DVT); catheter-related infections (CRI), and postoperative sepsis (PS). Empirical Bayes statistic (EB) was used to estimate whether the variation was systematic; logistic-multilevel modelling determined what proportion of the variation was explained by the hospital; and, shrunken residuals, as provided by multilevel modelling, were plotted to flag hospitals performing worse than expected. Variation across hospitals was observed to be systematic in all indicators, with EB values ranging from 0.19 (CI95%:0.12 to 0.28) in PE-DVT to 0.34 (CI95%:0.25 to 0.45) in DU. A significant proportion of the variance was explained by the hospital, once patient case-mix was adjusted: from a 6% in MLM (CI95%:3% to 11%) to a 24% (CI95%:20% to 30%) in CRI. All PSI were able to flag hospitals with rates over the expected, although this capacity decreased when the largest hospitals were analysed. Five PSI showed reasonable empirical properties to screen healthcare performance in Spanish hospitals, particularly in the largest ones.
    Full-text · Article · Feb 2012 · BMC Medical Research Methodology
  • Source
    • "Consistent with previous work [14,15,24], we evaluated the effects of potentially confounding factors, i.e., race, insurance status, hospital surgical volume, obesity, bi-laterality (two primary procedures during the same surgery), chronic kidney disease, and hypercoagulable state (including cancer and genetic predisposition), by adding them, one at a time, to the crude model (with comorbidity exposure, age group, and sex). If the addition of a given factor did not change the odds ratios for the comorbidity exposure by more than 10%, it was deleted from the final model. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Venous thromboembolism is a common, fatal, and costly injury which complicates major surgery in older adults. The American College of Chest Physicians recommends high potency prophylaxis regimens for individuals undergoing total hip or knee replacement (THR or TKR), but surgeons are reluctant to prescribe them due to fear of excess bleeding. Identifying a high risk cohort such as older adults with comorbidities and co-occurring comorbidities who might benefit most from high potency prophylaxis would improve how we currently perform preoperative assessment. Using the Nationwide Inpatient Sample, we identified older adults who underwent THR or TKR in the U.S. between 2003 and 2006. Our outcome was VTE, including any pulmonary embolus or deep venous thrombosis. We performed multivariate logistic regression analyses to assess the effects of comorbidities on VTE occurrence. Comorbidities under consideration included coronary artery disease, congestive heart failure (CHF), chronic obstructive pulmonary disease (COPD), diabetes, and cerebrovascular disease. We also examined the impact of co-occurring comorbidities on VTE rates. CHF increased odds of VTE in both the THR cohort (OR = 3.08 95% CI 2.05-4.65) and TKR cohort (OR = 2.47 95% CI 1.95-3.14). COPD led to a 50% increase in odds in the TKR cohort (OR = 1.49 95% CI 1.31-1.70). The data did not support synergistic effect of co-occurring comorbidities with respect to VTE occurrence. Older adults with CHF undergoing THR or TKR and with COPD undergoing TKR are at increased risk of VTE. If confirmed in other datasets, these older adults may benefit from higher potency prophylaxis.
    Full-text · Article · Sep 2010 · BMC Geriatrics
Show more