Toward Robust Information: Data Quality and Inter-Rater Reliability in the American College of Surgeons National Surgical Quality Improvement Program

Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL 60611, USA.
Journal of the American College of Surgeons (Impact Factor: 4.45). 01/2010; 210(1):6-16. DOI: 10.1016/j.jamcollsurg.2009.09.031
Source: PubMed

ABSTRACT Data used for evaluating quality of medical care need to be of high reliability to ensure valid quality assessment and benchmarking. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has continually emphasized the collection of highly reliable clinical data through its program infrastructure.
We provide a detailed description of the various mechanisms used in ACS NSQIP to assure collection of high quality data, including training of data collectors (surgical clinical reviewers) and ongoing audits of data reliability. For the 2005 through 2008 calendar years, inter-rater reliability was calculated overall and for individual variables using percentages of agreement between the data collector and the auditor. Variables with > 5% disagreement are flagged for educational efforts to improve accurate collection. Cohen's kappa was estimated for selected variables from the 2007 audit year.
Inter-rater reliability audits show that overall disagreement rates on variables have fallen from 3.15% in 2005 (the first year of public enrollment in ACS NSQIP) to 1.56% in 2008. In addition, disagreement levels for individual variables have continually improved, with 26 individual variables demonstrating > 5% disagreement in 2005, to only 2 such variables in 2008. Estimated kappa values suggest substantial or almost perfect agreement for most variables.
The ACS NSQIP has implemented training and audit procedures for its hospital participants that are highly effective in collecting robust data. Audit results show that data have been reliable since the program's inception and that reliability has improved every year.


Available from: Karen Richards, Jul 08, 2014
1 Follower
  • Source
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Surgical site infections (SSIs) result in significant patient morbidity following immediate tissue expander breast reconstruction (ITEBR). This study determined a single institution's 30-day SSI rate and benchmarked it against that among national institutions participating in the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP). Women who underwent ITEBR with/without acellular dermal matrix (ADM) were identified using the ACS-NSQIP database between 2005 and 2011. Patient characteristics associated with the 30-day SSI rate were determined, and differences in rates between our institution and the national database were assessed. 12,163 patients underwent ITEBR, including 263 at our institution. SSIs occurred in 416 (3.4%) patients nationwide excluding our institution, with lower rates observed at our institution (1.9%). Nationwide, SSIs were significantly more common in ITEBR patients with ADM (4.5%) compared to non-ADM patients (3.2%, P=0.005), and this trend was observed at our institution (2.1% vs. 1.6%, P=1.00). A multivariable analysis of all institutions identified age ≥50 years (odds ratio [OR], 1.4; confidence interval [CI], 1.1-1.7), body mass index ≥30 kg/m(2) vs. <25 kg/m(2) (OR, 3.4; CI, 2.6-4.5), and operative time >4.25 hours (OR, 1.9; CI, 1.5-2.4) as risk factors for SSIs. Our institutional SSI rate was lower than the nationwide rate (OR, 0.4; CI, 0.2-1.1), although this difference was not statistically significant (P=0.07). The 30-day SSI rate at our institution in patients who underwent ITEBR was lower than the nation. SSIs occurred more frequently in procedures involving ADM both nationally and at our institution.
    03/2015; 42(2):194-200. DOI:10.5999/aps.2015.42.2.194
  • [Show abstract] [Hide abstract]
    ABSTRACT: Many orthopaedic surgical procedures can be performed with either regional or general anesthesia. We hypothesized that total hip arthroplasty with regional anesthesia is associated with less postoperative morbidity and mortality than total hip arthroplasty with general anesthesia. This retrospective propensity-matched cohort study utilizing the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) database included patients who had undergone total hip arthroplasty from 2007 through 2011. After matching, logistic regression was used to determine the association between the type of anesthesia and deep surgical site infections, hospital length of stay, thirty-day mortality, and cardiovascular and pulmonary complications. Of 12,929 surgical procedures, 5103 (39.5%) were performed with regional anesthesia. The adjusted odds for deep surgical site infections were significantly lower in the regional anesthesia group than in the general anesthesia group (odds ratio [OR] = 0.38; 95% confidence interval [CI] = 0.20 to 0.72; p < 0.01). The hospital length of stay (geometric mean) was decreased by 5% (95% CI = 3% to 7%; p < 0.001) with regional anesthesia, which translates to 0.17 day for each total hip arthroplasty. Regional anesthesia was also associated with a 27% decrease in the odds of prolonged hospitalization (OR = 0.73; 95% CI = 0.68 to 0.89; p < 0.001). The mortality rate was not significantly lower with regional anesthesia (OR = 0.78; 95% CI = 0.43 to 1.42; p > 0.05). The adjusted odds for cardiovascular complications (OR = 0.61; 95% CI = 0.44 to 0.85) and respiratory complications (OR = 0.51; 95% CI = 0.33 to 0.81) were all lower in the regional anesthesia group. Compared with general anesthesia, regional anesthesia for total hip arthroplasty was associated with a reduction in deep surgical site infection rates, hospital length of stay, and rates of postoperative cardiovascular and pulmonary complications. These findings could have an important medical and economic impact on health-care practice. Therapeutic Level III. See Instructions for Authors for a complete description of levels of evidence. Copyright © 2015 by The Journal of Bone and Joint Surgery, Incorporated.
    The Journal of Bone and Joint Surgery 02/2015; 97(3):186-193. DOI:10.2106/JBJS.N.00612 · 4.31 Impact Factor