Pencina, M. J. , D'Agostino, R. B. , Sr, D'Agostino, R. B. Jr , & Vasan, R. S. Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat. Med. 27, 157-172

Department of Mathematics and Statistics, Framingham Heart Study, Boston University, Boston, MA 02215, USA.
Statistics in Medicine (Impact Factor: 1.83). 01/2008; 27(2):157-72; discussion 207-12. DOI: 10.1002/sim.2929
Source: PubMed


Identification of key factors associated with the risk of developing cardiovascular disease and quantification of this risk using multivariable prediction algorithms are among the major advances made in preventive cardiology and cardiovascular epidemiology in the 20th century. The ongoing discovery of new risk markers by scientists presents opportunities and challenges for statisticians and clinicians to evaluate these biomarkers and to develop new risk formulations that incorporate them. One of the key questions is how best to assess and quantify the improvement in risk prediction offered by these new models. Demonstration of a statistically significant association of a new biomarker with cardiovascular risk is not enough. Some researchers have advanced that the improvement in the area under the receiver-operating-characteristic curve (AUC) should be the main criterion, whereas others argue that better measures of performance of prediction models are needed. In this paper, we address this question by introducing two new measures, one based on integrated sensitivity and specificity and the other on reclassification tables. These new measures offer incremental information over the AUC. We discuss the properties of these new measures and contrast them with the AUC. We also develop simple asymptotic tests of significance. We illustrate the use of these measures with an example from the Framingham Heart Study. We propose that scientists consider these types of measures in addition to the AUC when assessing the performance of newer biomarkers.

35 Reads
    • "To compare the predictive values of the different risk scores on cognition, we used the area under the receiver operating characteristic curves (AUC) [30]. AUC comparisons were fit using the SAS % add_predictive macro. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background: Research concerning the link between individual vascular risk factors and cognition is plentiful but few studies have investigated the role of global vascular risk. We examined the cross-time associations of several vascular risk scores with cognitive performance during aging. Methods: Using data from the French SU.VI.MAX cohort, we studied a sample of 3061 participants. Framingham coronary heart disease, cardiovascular and stroke risk profiles were computed using baseline data (1994-1996). Cognitive performance was assessed after a mean of 13 years via a battery of six validated instruments. Principal component analysis identified scores for verbal memory and working memory. Associations between risk profiles (as continuous variables and in quartiles (Q)) and subsequent poor performance (defined as cognitive score ≤10th percentile) were examined via logistic regression (odds ratios, 95% CI) and analysis of covariance. Results: All continuous-scale Framingham risk scores assessed at midlife were inversely and uniformly associated with subsequent poor global cognitive performance, and especially in terms of verbal memory. Considering risk score Q, higher Q were associated with poorer performance in verbal memory: The fully-adjusted odds ratios (95% CI), comparing Q4 versus Q1, were 2.84 (1.70, 4.75), 2.31 (1.43, 3.73) and 1.77 (1.13, 2.76) for Framingham coronary heart disease, cardiovascular and stroke risk profiles, respectively. Similar findings were observed when modeling cognitive outcomes as continuous variables using covariance analyses. Conclusion: This study supports the existence of an inverse cross-time association between midlife vascular risk profiles and subsequent poor cognitive performance, especially in the verbal memory domain. Beyond their importance as regards vascular risk, such risk scores may help primary prevention efforts in identifying and targeting middle-aged individuals at high risk of cognitive aging.
    Atherosclerosis 09/2015; 243(1):286-292. DOI:10.1016/j.atherosclerosis.2015.09.023 · 3.99 Impact Factor
  • Source
    • "The net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) were calculated using the software R (Ver. 2.12.1) to examine if the Arg/ADMA ratio may improve the prediction of pathological increase of IMT, compared to the basal model as well as the model including ADMA [27] "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background: Asymmetric dimethylarginine (ADMA), which acts an endogenous inhibitor of nitric oxide synthase (NOS), is involved in the pathogenesis of cardiovascular disease. Arginine (Arg) may regulate vascular endothelial function, since Arg is the substrate of NO competing with ADMA. In our previous study, low Arg/ADMA ratio is an independent risk for microangiopathy-related cerebral damage. Purpose: Here, we performed a cross-sectional study to evaluate the association between the Arg/ADMA ratio and the maximal intima-media thickness (IMT) in the carotid artery. Subjects and Methods: Participants were 785 community-dwelling Japanese people without any severe disorders. Plasma concentration of Arg and ADMA in fasting blood sample was determined using HPLC. IMT was measured in the bilateral carotid artery by ultrasonography. Results: Among quartiles stratified by the Arg/ADMA ratio, ANOVA showed a significant difference in IMT and the IMT in Q1 (the lowest quartile) was significantly higher than that in Q4 (the highest quartile). In multiple linear regression analysis, age, the male gender, lower BMI, the presence of hypertension and lower Arg/ADMA ratio were independently correlated with IMT, while IMT was not correlated with Arg or ADMA alone. In addition, the Arg/ADMA ratio was associated with IMT independent of age, sex, BMI and the presence of hypertension with odds ratio 0.21 (95%CI: 0.05–0.88) in multiple logistic regression analysis for IMT 1.5 mm or more. Conclusion: Imbalance of Arg and ADMA is independently involved in the progression of atherosclerosis, and the Arg/ADMA ratio may be a sensitive marker for atherosclerosis.
    Atherosclerosis 03/2015; 239(1). DOI:10.1016/j.atherosclerosis.2014.12.030 · 3.99 Impact Factor
  • Source
    • "curring. It is related to the difference in sensitivities and one minus the specificities between the Cox PH model with and without the BP variation measure [17]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Recent data have highlighted shortcomings of the usual blood pressure (BP) hypothesis in several populations, and emphasized the importance of visit-to-visit variability of BP in predicting cardiovascular events. Herein, we aimed at assessing the association between visit-to-visit BP variability and outcomes in chronic heart failure (CHF) patients enrolled in the Heart failure Endpoint evaluation of Angiotensin II Antagonist Losartan (HEAAL). The HEAAL study randomized 3834 patients with HF and reduced ejection fraction administered 150mg or 50mg losartan daily in a double blind, randomized, controlled trial. The patients were followed up for up to 6.8years after randomization, and BP was measured at 3 time points in the first year and at semi-annual visits in the years thereafter. Three measures of visit-to-visit BP variability were computed for each subject: the standard deviation, the coefficient of variation and the average absolute visit-to-visit variation. Cox proportional hazard models were used to investigate the relationship between variations in systolic blood pressure, baseline covariates and the time to death or heart failure hospitalization (i.e. primary outcome). In multivariate analyses stratified on baseline BP, the patients with higher visit-to visit BP variability exhibited poorer outcomes (average absolute difference in SBP in mmHg:hazard ratio: 1.023 [95% CI (1.013, 1.034), P<0.0001]), independent from high dose losartan (still beneficial). For the first time, visit-to-visit BP variability was found elevated in CHF patients with reduced ejection fraction, and associated with poorer cardiovascular outcomes. Such assessments should be prioritized for testing prevention strategies in CHF. This study is registered with the, number NCT00090259. Copyright © 2015 Elsevier Ireland Ltd. All rights reserved.
    International journal of cardiology 03/2015; 187(1):183-189. DOI:10.1016/j.ijcard.2015.03.169 · 4.04 Impact Factor
Show more

Preview (2 Sources)

35 Reads
Available from