Toward robust information: data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program.

Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL 60611, USA.
Journal of the American College of Surgeons (Impact Factor: 4.45). 01/2010; 210(1):6-16. DOI: 10.1016/j.jamcollsurg.2009.09.031
Source: PubMed

ABSTRACT Data used for evaluating quality of medical care need to be of high reliability to ensure valid quality assessment and benchmarking. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has continually emphasized the collection of highly reliable clinical data through its program infrastructure.
We provide a detailed description of the various mechanisms used in ACS NSQIP to assure collection of high quality data, including training of data collectors (surgical clinical reviewers) and ongoing audits of data reliability. For the 2005 through 2008 calendar years, inter-rater reliability was calculated overall and for individual variables using percentages of agreement between the data collector and the auditor. Variables with > 5% disagreement are flagged for educational efforts to improve accurate collection. Cohen's kappa was estimated for selected variables from the 2007 audit year.
Inter-rater reliability audits show that overall disagreement rates on variables have fallen from 3.15% in 2005 (the first year of public enrollment in ACS NSQIP) to 1.56% in 2008. In addition, disagreement levels for individual variables have continually improved, with 26 individual variables demonstrating > 5% disagreement in 2005, to only 2 such variables in 2008. Estimated kappa values suggest substantial or almost perfect agreement for most variables.
The ACS NSQIP has implemented training and audit procedures for its hospital participants that are highly effective in collecting robust data. Audit results show that data have been reliable since the program's inception and that reliability has improved every year.

1 Bookmark
  • [Show abstract] [Hide abstract]
    ABSTRACT: Cardiac complications are a major cause of postoperative morbidity. The purpose of this study was to determine the rates, risk factors, and time of occurrence for cardiac complications within thirty days after primary unilateral total knee arthroplasty and total hip arthroplasty. The American College of Surgeons National Surgical Quality Improvement Program data set from 2006 to 2011 was used to identify all total knee arthroplasties and total hip arthroplasties. Cardiac complications occurring within thirty days after surgery were the primary outcome measure. Patients were designated as having a history of cardiac disease if they had a new diagnosis or exacerbation of chronic congestive heart failure or a history of angina within thirty days before surgery, a history of myocardial infarction within six months, and/or any percutaneous cardiac intervention or other major cardiac surgery at any time. An analysis of the occurrence of all major cardiac complications and deaths within the thirty-day postoperative time frame was performed. For the 46,322 patients managed with total knee arthroplasty or total hip arthroplasty, the cardiac complication rate was 0.33% (n = 153) at thirty days postoperatively. In both the total knee arthroplasty and total hip arthroplasty groups, an age of eighty years or more (odds ratios [ORs] = 27.95 and 3.72), hypertension requiring medication (ORs = 4.74 and 2.59), and a history of cardiac disease (ORs = 4.46 and 2.80) were the three most significant predictors for the development of postoperative cardiac complications. Of the patients with a cardiac complication, the time of occurrence was within seven days after surgery for 79% (129 of the 164 patients for whom the time of occurrence could be determined). An age of eighty years or more, a history of cardiac disease, and hypertension requiring medication are significant risk factors for developing postoperative cardiac complications following primary unilateral total knee arthroplasty and total hip arthroplasty. Consideration should be given to a preoperative cardiology evaluation and co-management in the perioperative period for individuals with these risk factors. Prognostic Level III. See Instructions for Authors for a complete description of levels of evidence. Copyright © 2014 by The Journal of Bone and Joint Surgery, Incorporated.
    The Journal of Bone and Joint Surgery 12/2014; 96(24):2025-31. · 4.31 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Health care is evolving into a value-based reimbursement system focused on quality and outcomes. Reported outcomes from national databases are used for quality improvement projects and public reporting. This study compared reported outcomes in cardiac and thoracic surgery from two validated reporting databases-the Society of Thoracic Surgeons (STS) database and the National Surgical Quality Improvement Program (NSQIP)-from January 2011 to June 2012. Quality metrics and outcomes included mortality, wound infection, prolonged ventilation, pneumonia, renal failure, stroke, and cardiac arrest. Comparison was made by chi-square analysis. A total of 737 and 177 cardiac surgery cases and 451 and 105 thoracic surgery cases were captured by the STS database and NSQIP, respectively. Within cardiac surgery, there was a statistically significant difference in the reported rates of prolonged ventilation, renal failure, and mortality. No significant differences were found for the thoracic surgery data. In conclusion, our data indicated a significant discordance in quality reporting for cardiac surgery between the NSQIP and the STS databases. The disparity between databases and duplicate participation strongly indicates that a unified national quality reporting program is required. Consolidation of reporting databases and standardization of morbidity definitions across all databases may improve participation and reduce hospital cost.
    Proceedings (Baylor University. Medical Center) 01/2015; 28(1):14-7.
  • [Show abstract] [Hide abstract]
    ABSTRACT: Postoperative occurrences have been associated with an increased risk of readmission, yet these occurrences and their timing have not been well characterized.
    Diseases of the Colon & Rectum 11/2014; 57(11):1309-1316. · 3.20 Impact Factor

Full-text (2 Sources)

Available from
Jul 9, 2014