How best to measure surgical quality? Comparison of the Agency for Healthcare Research and Quality Patient Safety Indicators (AHRQ-PSI) and the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) postoperative adverse events at a single institution

Department of Surgery, Mayo Clinic College of Medicine, Rochester, MN 55905, USA.
Surgery (Impact Factor: 3.38). 08/2011; 150(5):943-9. DOI: 10.1016/j.surg.2011.06.020
Source: PubMed


Evaluating surgical outcomes is an important tool to compare providers and institutions and to drive process improvements. Differing methodologies, however, may provide conflicting measurements of similar clinical outcomes making comparisons difficult. ACS-NSQIP is a validated, risk-adjusted, clinically derived data methodology to compare observed to expected outcomes after a wide variety of operations. The AHRQ-PSI are a set of computer algorithms to identify potential adverse in-patient events using secondary ICD-9-CM diagnosis and procedure codes from hospital discharge abstracts.
We compared the ACS-NSQIP and AHRQ-PSI methods for hospital general surgical (n = 6565) or vascular surgical inpatients procedures (n = 1041) at a tertiary-care academic institution from April 2006 to June 2009 on 7 adverse event types.
ACS-NSQIP inpatient adverse events were identified in 564 (7.4%) patients. AHRQ-PSIs were identified in 268 (3.5%) patients. Only 159 (2.1%) patients had inpatient events identified by both methods. Using ACS-NSQIP as the clinically based standard the sensitivity of the specific AHRQ-PSI ranged from 0.030 for infections to 0.535 for PE/DVT. Positive predictive values of AHRQ-PSI ranged from 18% for hemorrhage/hematoma to 89% for renal failure. Greater agreement at greater ASA class and wound classification was observed.
AHRQ-PSI algorithms identified less than a third of the ACS-NSQIP clinically important adverse events. Furthermore, the AHRQ-PSI identified a large number of events with no corresponding clinically important adverse outcomes. The sensitivity of the AHRQ-PSI for detecting clinically relevant adverse events identified by the ACS-NSQIP varied widely. The AHRQ-PSI as applied to postoperative patients is a poor measure of quality performance.

Download full-text


Available from: Robert Cima, Sep 17, 2014
241 Reads
  • Source
    • "However, many national adverse event reporting initiatives -- such as those undertaken by the National Quality Forum, the Agency for Healthcare Research and Quality, the Joint Commission, and the National Surgical Quality Improvement Program -- rely on a relatively small set of common indicators and thus lack the richness, detail, and variety that can often be obtained through use of a customized hospital-based ERS of the type utilized in this study [20-22]. Surgeons need to pay attention to the data that can be obtained by the use of facility or system-specific event reporting systems. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Many health care facilities have developed electronic reporting systems for identifying and reporting adverse events (AEs), so that measures can be taken to improve patient safety. Although several studies have examined AEs in surgical settings, there has not previously been a systematic assessment of the variations in adverse event rates among different types of surgery, nor an identification of the particular types of AEs that are most common within each surgical category. Additionally, this study will identify the AE severity level associated with each of the AE category types. Methods This retrospective observational study was conducted at three Midwestern hospitals that are part of a large integrated healthcare system. Data from 2006 through 2009 were analyzed to determine the rates of reported adverse events (per 1,000 hospitalizations involving a surgical procedure) for 96 categories of surgery as classified according to the ICD-9-CM procedural coding system. Univariate and bivariate summary statistics were compiled for AEs by type, severity, and patient age. Results During the four-year study period, there was a total of 82,784 distinct hospitalizations involving at least one surgical procedure at these three hospitals. At least one adverse event was reported at 5,368 (6.5%) of those hospitalizations. The mean rate of AEs among all surgical procedure groups was 82.8 AEs per 1,000 hospitalizations. Adverse event rates varied widely among surgical categories with a high of 556.7 AEs per 1,000 hospitalizations for operations on the heart and pericardium. The most common type of adverse event involved care management, followed by medication events and events related to invasive procedures. Conclusions Detecting variations in AEs among surgical categories can be useful for surgeons and for hospital quality assurance personnel. Documenting the specific AE incidence rates among the most common types of surgical categories, and determining AE severity and age distributions within surgical categories will enable officials to better identify specific patient safety needs and develop appropriately targeted interventions for improvement.
    Patient Safety in Surgery 05/2014; 8(1):23. DOI:10.1186/1754-9493-8-23
  • Source
    • "Risk-adjusted morbidity and mortality can be determined, and quality improvement projects can be developed from the NSQIP database. Since its initiation, it has become the standard for evaluation of surgical outcomes and is anticipated to play a large role in future Medicare pay for performance initiatives [2]. In October 2008, NSQIP was initiated in four pediatric hospitals [3]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background: In October 2008, the American College of Surgeons revealed the National Surgical Quality Improvement Program (NSQIP) Pediatric in an effort to improve quality of surgical care in children. A 5% disagreement rate of data reported between institutions is accepted. The two goals of this study were to (1) determine if the random sampling performed with NSQIP data collection was representative of the population, and (2) verify that data captured in NSQIP was accurate. Methods: For children undergoing laparoscopic appendectomy from April 2010-April 2011, demographic data, length of stay (LOS), and rates of surgical site infection (SSI) and postoperative abscess recorded in NSQIP (group 1) were compared with data from chart review (group 2). Secondarily, all NSQIP data were examined for accuracy by comparing relevant data points to existing databases. All disagreements were further examined with review of the medical chart. Unpaired t-test and χ(2) with Fisher's exact test were used in the statistical analysis. Results: NSQIP Pediatric captured data from 126 children (group 1); group 2 had 525 children. There were no significant differences in age, body mass index, gender, race or LOS between the two groups. Rate of SSI was 1.6% in group 1 and 1.7% in group 2 (P = 0.92). Abscess rate was 1.6% in group 1 and 3.4% in group 2 (P = 0.28). There were six errors in the NSQIP database. One child was listed as having two SSI. One child with postoperative abscess was missed and another was not counted as they were not categorized correctly. Recorded LOS was incorrect for two children and the other had incorrect age. Conclusions: NSQIP Pediatric captured a representative sample of patients undergoing laparoscopic appendectomy. Errors were found in the reporting of outcomes for SSI and postoperative abscess in children undergoing laparoscopic appendectomy. Given the low incidence of these outcomes, there is little effect on percentages of complications reported.
    Journal of Surgical Research 06/2013; 179(2). DOI:10.1016/j.jss.2013.05.066 · 1.94 Impact Factor
  • Source
    • "The input of data is labour intensive, and the results are only as good as the data input. Furthermore, the results are based on interpretation of data in specific categories, thus missing complications that do not fall into these specific areas [52-54]. This ACS-NSQIP programme is also building up a large database of information that should hopefully produce more effective risk stratification scores in the future. "
    [Show abstract] [Hide abstract]
    ABSTRACT: There are a vast number of operations carried out every year, with a small proportion of patients being at highest risk of mortality and morbidity. There has been considerable work to try and identify these high-risk patients. In this paper, we look in detail at the commonly used perioperative risk prediction models. Finally, we will be looking at the evolution and evidence for functional assessment and the National Surgical Quality Improvement Program (in the USA), both topical and exciting areas of perioperative prediction.
    Critical care (London, England) 05/2013; 17(3):226. DOI:10.1186/cc11904 · 4.48 Impact Factor
Show more