Performance of reclassification statistics in comparing risk prediction models

Division of Preventive Medicine, Brigham and Women's Hospital, Boston, MA, USA.
Biometrical Journal (Impact Factor: 0.95). 03/2011; 53(2):237-58. DOI: 10.1002/bimj.201000078
Source: PubMed


Concerns have been raised about the use of traditional measures of model fit in evaluating risk prediction models for clinical use, and reclassification tables have been suggested as an alternative means of assessing the clinical utility of a model. Several measures based on the table have been proposed, including the reclassification calibration (RC) statistic, the net reclassification improvement (NRI), and the integrated discrimination improvement (IDI), but the performance of these in practical settings has not been fully examined. We used simulations to estimate the type I error and power for these statistics in a number of scenarios, as well as the impact of the number and type of categories, when adding a new marker to an established or reference model. The type I error was found to be reasonable in most settings, and power was highest for the IDI, which was similar to the test of association. The relative power of the RC statistic, a test of calibration, and the NRI, a test of discrimination, varied depending on the model assumptions. These tools provide unique but complementary information.

Full-text preview

Available from:
  • Source
    • "Models were tested using the c-index improvement, integrated discrimination improvement (IDI) and net reclassification improvement (NRI) according to recently suggested risk limits 0% to <5%, 5% to <10%, 10% to <20% and ≥20% for 10 year risk limits [29]. In addition, ‘Clinical’ NRI quantifies the improvement in prediction in the intermediate risk group (5–20%) which incorporates a correction for the expected value of improvement [30]. This test measures the reclassification where only individuals in the intermediate group are tested with the GRS and have their risk recalculated. "
    [Show abstract] [Hide abstract]
    ABSTRACT: More accurate coronary heart disease (CHD) prediction, specifically in middle-aged men, is needed to reduce the burden of disease more effectively. We hypothesised that a multilocus genetic risk score could refine CHD prediction beyond classic risk scores and obtain more precise risk estimates using a prospective cohort design. Using data from nine prospective European cohorts, including 26,221 men, we selected in a case-cohort setting 4,818 healthy men at baseline, and used Cox proportional hazards models to examine associations between CHD and risk scores based on genetic variants representing 13 genomic regions. Over follow-up (range: 5-18 years), 1,736 incident CHD events occurred. Genetic risk scores were validated in men with at least 10 years of follow-up (632 cases, 1361 non-cases). Genetic risk score 1 (GRS1) combined 11 SNPs and two haplotypes, with effect estimates from previous genome-wide association studies. GRS2 combined 11 SNPs plus 4 SNPs from the haplotypes with coefficients estimated from these prospective cohorts using 10-fold cross-validation. Scores were added to a model adjusted for classic risk factors comprising the Framingham risk score and 10-year risks were derived. Both scores improved net reclassification (NRI) over the Framingham score (7.5%, p = 0.017 for GRS1, 6.5%, p = 0.044 for GRS2) but GRS2 also improved discrimination (c-index improvement 1.11%, p = 0.048). Subgroup analysis on men aged 50-59 (436 cases, 603 non-cases) improved net reclassification for GRS1 (13.8%) and GRS2 (12.5%). Net reclassification improvement remained significant for both scores when family history of CHD was added to the baseline model for this male subgroup improving prediction of early onset CHD events. Genetic risk scores add precision to risk estimates for CHD and improve prediction beyond classic risk factors, particularly for middle aged men.
    Full-text · Article · Jul 2012 · PLoS ONE
  • Source
    • "The clinical utility of risk models can also be estimated by measures of reclassification. The net reclassification index (NRI) and the integrated discrimination improvement (IDI) assess whether a novel marker can improve the risk classification for individuals, as compared with models without the marker of interest (Cook & Paynter, 2011; Steyerberg et al. 2010). Studies incorporating such analyses will also be needed to determine whether a genetic risk score provides any additional clinical utility "
    [Show abstract] [Hide abstract]
    ABSTRACT: A less favorable cardiovascular risk factor profile but paradoxically lower cardiovascular morbidity and mortality have been observed in Hispanics--a pattern often referred to as the Hispanic paradox. It has been proposed that the specific genetic susceptibility of this admixed population and gene-environment interactions may partly explain the paradox. During the past few years, there have been major advances in the identification of genetic risk factors using genome-wide association studies (GWAS) for cardiovascular disease, especially in Caucasians. However, no GWAS of cardiovascular disease have been reported in Hispanics. In the Costa Rican Heart Study, we reported both the consistency and disparity of genetic effects on risk of coronary heart disease (CHD) between Hispanics and other ethnic groups. We demonstrated that the improvement in the identified genetic markers on discrimination of CHD in Hispanics was modest. Future genetic research on Hispanics should consider the diversity in genetic structure, lifestyle, and socioeconomics among various subpopulations and comprehensively evaluate potential gene-environment interactions in relation to cardiovascular risk.
    Preview · Article · Jan 2011 · Trends in cardiovascular medicine

  • No preview · Article · May 2011 · International journal of cardiology
Show more