Pencina, M. J. , D'Agostino, R. B. , Sr, D'Agostino, R. B. Jr , & Vasan, R. S. Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat. Med. 27, 157-172

Department of Mathematics and Statistics, Framingham Heart Study, Boston University, Boston, MA 02215, USA.
Statistics in Medicine (Impact Factor: 1.83). 01/2008; 27(2):157-72; discussion 207-12. DOI: 10.1002/sim.2929
Source: PubMed

ABSTRACT Identification of key factors associated with the risk of developing cardiovascular disease and quantification of this risk using multivariable prediction algorithms are among the major advances made in preventive cardiology and cardiovascular epidemiology in the 20th century. The ongoing discovery of new risk markers by scientists presents opportunities and challenges for statisticians and clinicians to evaluate these biomarkers and to develop new risk formulations that incorporate them. One of the key questions is how best to assess and quantify the improvement in risk prediction offered by these new models. Demonstration of a statistically significant association of a new biomarker with cardiovascular risk is not enough. Some researchers have advanced that the improvement in the area under the receiver-operating-characteristic curve (AUC) should be the main criterion, whereas others argue that better measures of performance of prediction models are needed. In this paper, we address this question by introducing two new measures, one based on integrated sensitivity and specificity and the other on reclassification tables. These new measures offer incremental information over the AUC. We discuss the properties of these new measures and contrast them with the AUC. We also develop simple asymptotic tests of significance. We illustrate the use of these measures with an example from the Framingham Heart Study. We propose that scientists consider these types of measures in addition to the AUC when assessing the performance of newer biomarkers.

28 Reads
  • Source
    • "The net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) were calculated using the software R (Ver. 2.12.1) to examine if the Arg/ADMA ratio may improve the prediction of pathological increase of IMT, compared to the basal model as well as the model including ADMA [27] "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background: Asymmetric dimethylarginine (ADMA), which acts an endogenous inhibitor of nitric oxide synthase (NOS), is involved in the pathogenesis of cardiovascular disease. Arginine (Arg) may regulate vascular endothelial function, since Arg is the substrate of NO competing with ADMA. In our previous study, low Arg/ADMA ratio is an independent risk for microangiopathy-related cerebral damage. Purpose: Here, we performed a cross-sectional study to evaluate the association between the Arg/ADMA ratio and the maximal intima-media thickness (IMT) in the carotid artery. Subjects and Methods: Participants were 785 community-dwelling Japanese people without any severe disorders. Plasma concentration of Arg and ADMA in fasting blood sample was determined using HPLC. IMT was measured in the bilateral carotid artery by ultrasonography. Results: Among quartiles stratified by the Arg/ADMA ratio, ANOVA showed a significant difference in IMT and the IMT in Q1 (the lowest quartile) was significantly higher than that in Q4 (the highest quartile). In multiple linear regression analysis, age, the male gender, lower BMI, the presence of hypertension and lower Arg/ADMA ratio were independently correlated with IMT, while IMT was not correlated with Arg or ADMA alone. In addition, the Arg/ADMA ratio was associated with IMT independent of age, sex, BMI and the presence of hypertension with odds ratio 0.21 (95%CI: 0.05–0.88) in multiple logistic regression analysis for IMT 1.5 mm or more. Conclusion: Imbalance of Arg and ADMA is independently involved in the progression of atherosclerosis, and the Arg/ADMA ratio may be a sensitive marker for atherosclerosis.
    Atherosclerosis 03/2015; 239(1). DOI:10.1016/j.atherosclerosis.2014.12.030 · 3.99 Impact Factor
  • Source
    • "curring. It is related to the difference in sensitivities and one minus the specificities between the Cox PH model with and without the BP variation measure [17]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Recent data have highlighted shortcomings of the usual blood pressure (BP) hypothesis in several populations, and emphasized the importance of visit-to-visit variability of BP in predicting cardiovascular events. Herein, we aimed at assessing the association between visit-to-visit BP variability and outcomes in chronic heart failure (CHF) patients enrolled in the Heart failure Endpoint evaluation of Angiotensin II Antagonist Losartan (HEAAL). The HEAAL study randomized 3834 patients with HF and reduced ejection fraction administered 150mg or 50mg losartan daily in a double blind, randomized, controlled trial. The patients were followed up for up to 6.8years after randomization, and BP was measured at 3 time points in the first year and at semi-annual visits in the years thereafter. Three measures of visit-to-visit BP variability were computed for each subject: the standard deviation, the coefficient of variation and the average absolute visit-to-visit variation. Cox proportional hazard models were used to investigate the relationship between variations in systolic blood pressure, baseline covariates and the time to death or heart failure hospitalization (i.e. primary outcome). In multivariate analyses stratified on baseline BP, the patients with higher visit-to visit BP variability exhibited poorer outcomes (average absolute difference in SBP in mmHg:hazard ratio: 1.023 [95% CI (1.013, 1.034), P<0.0001]), independent from high dose losartan (still beneficial). For the first time, visit-to-visit BP variability was found elevated in CHF patients with reduced ejection fraction, and associated with poorer cardiovascular outcomes. Such assessments should be prioritized for testing prevention strategies in CHF. This study is registered with the, number NCT00090259. Copyright © 2015 Elsevier Ireland Ltd. All rights reserved.
    International journal of cardiology 03/2015; 187(1):183-189. DOI:10.1016/j.ijcard.2015.03.169 · 4.04 Impact Factor
  • Source
    • "Finally, a new biomarker should ideally reclassify patients in more appropriate risk categories. We documented percentages of patients correctly reclassified (upwards or downwards) and, when available, the net reclassification index (NRI), which indicates the net number of patients reclassified to the correct risk category [8] [12]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background: Despite substantive growth in utilization of positron emission tomography (PET) myocardial perfusion imaging (MPI), evidence on its prognostic value is limited. We aimed to comprehensively evaluate the prognostic literature of PET perfusion measures according to the most recent American Heart Association recommendations for assessment of novel cardiovascular biomarkers. Methods: We searched the literature for studies reporting associations of PET MPI measures and outcomes in patients with known or suspected coronary artery disease. We documented hazard ratios (HR) and 95% confidence intervals (CI) of association effects and quantitatively synthesized them with random-effects meta-analyses. Discrimination, calibration and risk reclassification after addition of PET MPI measures to standard prognostic models were documented. Results: We identified 20 eligible studies with median n. = 551 patients. In meta-analyses, the extents of ischemic and scarred myocardium were significantly associated with cardiac death. Meta-analyses of multivariate estimates for abnormal summed stress score ≥. 4 and myocardial perfusion reserve <. 2 revealed significant associations with major adverse cardiovascular events [HR (95% CI) 2.30 (1.53-3.44) and 2.11 (1.33-3.36), respectively]. Changes in model discrimination, calibration or risk reclassification were reported in 5 studies (8 prognostic evaluations). There were marginal improvements in discrimination based on C index and no improvements in model calibration. Net reclassification index ranged from 9.8% to 14.2% and risk classification was significantly improved in 4/5 prognostic evaluations. Conclusions: PET MPI measures were strongly associated with adverse patient outcomes. Risk classification was more consistently improved than discrimination and calibration after addition of PET MPI measures, but reporting of such metrics was overall limited.
    03/2015; 6. DOI:10.1016/j.ijcha.2015.01.005
Show more


28 Reads
Available from