Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond

Department of Mathematics and Statistics, Framingham Heart Study, Boston University, Boston, MA 02215, USA.
Statistics in Medicine (Impact Factor: 2.04). 01/2008; 27(2):157-72; discussion 207-12. DOI: 10.1002/sim.2929
Source: PubMed

ABSTRACT Identification of key factors associated with the risk of developing cardiovascular disease and quantification of this risk using multivariable prediction algorithms are among the major advances made in preventive cardiology and cardiovascular epidemiology in the 20th century. The ongoing discovery of new risk markers by scientists presents opportunities and challenges for statisticians and clinicians to evaluate these biomarkers and to develop new risk formulations that incorporate them. One of the key questions is how best to assess and quantify the improvement in risk prediction offered by these new models. Demonstration of a statistically significant association of a new biomarker with cardiovascular risk is not enough. Some researchers have advanced that the improvement in the area under the receiver-operating-characteristic curve (AUC) should be the main criterion, whereas others argue that better measures of performance of prediction models are needed. In this paper, we address this question by introducing two new measures, one based on integrated sensitivity and specificity and the other on reclassification tables. These new measures offer incremental information over the AUC. We discuss the properties of these new measures and contrast them with the AUC. We also develop simple asymptotic tests of significance. We illustrate the use of these measures with an example from the Framingham Heart Study. We propose that scientists consider these types of measures in addition to the AUC when assessing the performance of newer biomarkers.

  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Background: Asymmetric dimethylarginine (ADMA), which acts an endogenous inhibitor of nitric oxide synthase (NOS), is involved in the pathogenesis of cardiovascular disease. Arginine (Arg) may regulate vascular endothelial function, since Arg is the substrate of NO competing with ADMA. In our previous study, low Arg/ADMA ratio is an independent risk for microangiopathy-related cerebral damage. Purpose: Here, we performed a cross-sectional study to evaluate the association between the Arg/ADMA ratio and the maximal intima-media thickness (IMT) in the carotid artery. Subjects and Methods: Participants were 785 community-dwelling Japanese people without any severe disorders. Plasma concentration of Arg and ADMA in fasting blood sample was determined using HPLC. IMT was measured in the bilateral carotid artery by ultrasonography. Results: Among quartiles stratified by the Arg/ADMA ratio, ANOVA showed a significant difference in IMT and the IMT in Q1 (the lowest quartile) was significantly higher than that in Q4 (the highest quartile). In multiple linear regression analysis, age, the male gender, lower BMI, the presence of hypertension and lower Arg/ADMA ratio were independently correlated with IMT, while IMT was not correlated with Arg or ADMA alone. In addition, the Arg/ADMA ratio was associated with IMT independent of age, sex, BMI and the presence of hypertension with odds ratio 0.21 (95%CI: 0.05–0.88) in multiple logistic regression analysis for IMT 1.5 mm or more. Conclusion: Imbalance of Arg and ADMA is independently involved in the progression of atherosclerosis, and the Arg/ADMA ratio may be a sensitive marker for atherosclerosis.
    Atherosclerosis 03/2015; 239(1). DOI:10.1016/j.atherosclerosis.2014.12.030 · 3.97 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Recent data have highlighted shortcomings of the usual blood pressure (BP) hypothesis in several populations, and emphasized the importance of visit-to-visit variability of BP in predicting cardiovascular events. Herein, we aimed at assessing the association between visit-to-visit BP variability and outcomes in chronic heart failure (CHF) patients enrolled in the Heart failure Endpoint evaluation of Angiotensin II Antagonist Losartan (HEAAL). The HEAAL study randomized 3834 patients with HF and reduced ejection fraction administered 150mg or 50mg losartan daily in a double blind, randomized, controlled trial. The patients were followed up for up to 6.8years after randomization, and BP was measured at 3 time points in the first year and at semi-annual visits in the years thereafter. Three measures of visit-to-visit BP variability were computed for each subject: the standard deviation, the coefficient of variation and the average absolute visit-to-visit variation. Cox proportional hazard models were used to investigate the relationship between variations in systolic blood pressure, baseline covariates and the time to death or heart failure hospitalization (i.e. primary outcome). In multivariate analyses stratified on baseline BP, the patients with higher visit-to visit BP variability exhibited poorer outcomes (average absolute difference in SBP in mmHg:hazard ratio: 1.023 [95% CI (1.013, 1.034), P<0.0001]), independent from high dose losartan (still beneficial). For the first time, visit-to-visit BP variability was found elevated in CHF patients with reduced ejection fraction, and associated with poorer cardiovascular outcomes. Such assessments should be prioritized for testing prevention strategies in CHF. This study is registered with the, number NCT00090259. Copyright © 2015 Elsevier Ireland Ltd. All rights reserved.
    International journal of cardiology 03/2015; 187:183-189. DOI:10.1016/j.ijcard.2015.03.169 · 6.18 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Rapidly assessing outcome in patients with acute decompensated heart failure is important but prognostic factors may differ from those used routinely for stable chronic heart failure. Multiple plasma biomarkers, besides the classic natriuretic peptides, have recently emerged as potential prognosticators. Furthermore, prognostic scores that combine clinical and biochemical data may also be useful. However, compared with the scores used in chronic heart failure, scores for acute decompensated heart failure have not been validated. This article reviews potential biomarkers, with a special focus on biochemical biomarkers, and possible prognostic scores that could be used by the clinician when assessing outcome in patients with acute heart failure. Copyright © 2014 Elsevier Masson SAS. All rights reserved.
    Archives of Cardiovascular Diseases 11/2014; 108(1). DOI:10.1016/j.acvd.2014.10.002 · 1.66 Impact Factor


Available from