Performance of reclassification statistics in comparing risk prediction models

Division of Preventive Medicine, Brigham and Women's Hospital, Boston, MA, USA.
Biometrical Journal (Impact Factor: 1.15). 03/2011; 53(2):237-58. DOI: 10.1002/bimj.201000078
Source: PubMed

ABSTRACT Concerns have been raised about the use of traditional measures of model fit in evaluating risk prediction models for clinical use, and reclassification tables have been suggested as an alternative means of assessing the clinical utility of a model. Several measures based on the table have been proposed, including the reclassification calibration (RC) statistic, the net reclassification improvement (NRI), and the integrated discrimination improvement (IDI), but the performance of these in practical settings has not been fully examined. We used simulations to estimate the type I error and power for these statistics in a number of scenarios, as well as the impact of the number and type of categories, when adding a new marker to an established or reference model. The type I error was found to be reasonable in most settings, and power was highest for the IDI, which was similar to the test of association. The relative power of the RC statistic, a test of calibration, and the NRI, a test of discrimination, varied depending on the model assumptions. These tools provide unique but complementary information.

  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from
    Annals of internal medicine 01/2015; 162(1):W1-W73. DOI:10.7326/M14-0698 · 16.10 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Left Ventricular Hypertrophy (LVH) is a major independent predictor of cardiovascular disease (CVD) survival, and is more prevalent in blacks than whites. In a large biracial population, we evaluated the ability of ECG-determined LVH (ECG-LVH) to reclassify CVD/coronary heart disease (CHD) events beyond traditional risk factors in blacks and whites. The analysis included 14,489 participants (mean age 54+/−5.7 years, 43.5% men, 26% black) from the Atherosclerosis Risk in Communities (ARIC) cohort, with baseline (1987–1989) ECG, followed for 10 years. Predicted risk for incident CVD and CHD were estimated using the 10-year Pooled Cohort and Framingham risk equations (base models 1a/1b), respectively. Models 2a and 2b included respective base model plus LVH by any of 10 traditional ECG-LVH criteria. Net reclassification improvement (NRI) was calculated, and the distribution of risk was compared using models 2a and 2b vs. models 1a and 1b, respectively. There were 792 (5.5%) 10-year Pooled Cohort CVD events, and 690 (4.8%) 10-year Framingham CHD events. LVH defined by any criteria was associated with CVD and CHD events [HR (95% CI): 1.62 (1.38-1.90) and 1.56 (1.32-1.86), respectively]. LVH did not significantly reclassify or improve C-statistic in models 2a/b [C-statistics: 0.767/0.719; NRI = 0.001 (p = NS)], compared with the base models 1a/b (C-statistics: 0.770/0.718), respectively. No racial interactions were observed. In this large cohort of black and white participants, ECG-LVH was associated with CVD/CHD risk, but did not significantly improve CVD and CHD events risk prediction beyond the new Pooled Cohort and most utilized Framingham risk equations in blacks or whites.
    American Heart Journal 01/2014; 169(1). DOI:10.1016/j.ahj.2014.09.013 · 4.56 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Recent studies have suggested that circulating levels of the tumor necrosis factor α receptor 1 (sTNFαR1) may be a useful predictor for the risk of end-stage renal disease (ESRD) in patients with diabetes. However, its potential utility as a biomarker has not been formally quantified.RESEARCH DESIGN AND METHODS: Circulating levels of sTNFαR1 were assessed in 429 patients with type 1 diabetes and overt nephropathy from the Finnish Diabetic Nephropathy (FinnDiane) cohort study. Predictors of incident ESRD over a median of 9.4 years of follow-up were determined by Cox regression and Fine-Gray competing risk analyses. The added value of sTNFαR1 was estimated via time-dependent receiver operating characteristic curves, net reclassification index (NRI), and integrated discrimination improvement (IDI) for survival data.RESULTS: A total of 130 individuals developed ESRD (28%; ESRD incidence rate of 3.4% per year). In cause-specific modeling, after adjusting for baseline renal status, predictors of increased incidence of ESRD in patients with overt nephropathy were an elevated HbA1c, shorter duration of diabetes, and circulating levels of sTNFαR1. Notably, sTNFαR1 outperformed estimated glomerular filtration rate in terms of R(2). Circulating levels of the sTNFαR1 also remained associated with ESRD after adjusting for the competing risk of death. A prediction model including sTNFαR1 (as a -0.5 fractional polynomial) was superior to a model without it, as demonstrated by better global fit, an increment of R(2), the C index, and area under the curve. Estimates of IDI and NRI(>0) were 0.22 (95% CI 0.16-0.28; P < 0.0001) and 0.98 (0.78-1.23; P < 0.0001), respectively. The median increment in the risk score after including sTNFαR1 in the prediction model was 0.18 (0.12-0.30; P < 0.0001).CONCLUSIONS: Circulating levels of sTNFαR1 are independently associated with the cumulative incidence of ESRD. This association is both significant and biologically plausible and appears to provide added value as a biomarker, based on the absolute values of NRI and IDI.
    Diabetes Care 05/2014; 37(8). DOI:10.2337/dc14-0225 · 8.57 Impact Factor


Available from