Pencina, M. J. , D'Agostino, R. B. , Sr, D'Agostino, R. B. Jr , & Vasan, R. S. Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat. Med. 27, 157-172

Department of Mathematics and Statistics, Framingham Heart Study, Boston University, Boston, MA 02215, USA.
Statistics in Medicine (Impact Factor: 1.83). 01/2008; 27(2):157-72; discussion 207-12. DOI: 10.1002/sim.2929
Source: PubMed


Identification of key factors associated with the risk of developing cardiovascular disease and quantification of this risk using multivariable prediction algorithms are among the major advances made in preventive cardiology and cardiovascular epidemiology in the 20th century. The ongoing discovery of new risk markers by scientists presents opportunities and challenges for statisticians and clinicians to evaluate these biomarkers and to develop new risk formulations that incorporate them. One of the key questions is how best to assess and quantify the improvement in risk prediction offered by these new models. Demonstration of a statistically significant association of a new biomarker with cardiovascular risk is not enough. Some researchers have advanced that the improvement in the area under the receiver-operating-characteristic curve (AUC) should be the main criterion, whereas others argue that better measures of performance of prediction models are needed. In this paper, we address this question by introducing two new measures, one based on integrated sensitivity and specificity and the other on reclassification tables. These new measures offer incremental information over the AUC. We discuss the properties of these new measures and contrast them with the AUC. We also develop simple asymptotic tests of significance. We illustrate the use of these measures with an example from the Framingham Heart Study. We propose that scientists consider these types of measures in addition to the AUC when assessing the performance of newer biomarkers.

  • Source
    • "The variables of í µí± < 0.10 in the univariable logistic analysis were analyzed by the multivariable analysis. The improvement by adding %abdominal VAT to discrimination and net reclassification of risks was assessed by comparing the area under the curve (AUC) of receiver operator characteristics and the estimation of both integrated discrimination improvement (IDI) and net reclassification improvement (NRI)[9]. All tests were two-tailed and í µí± values < 0.05 were considered statistically significant. "
    [Show abstract] [Hide abstract]
    ABSTRACT: . Aortic valve calcification (AVC) is a common feature of aging and is related to coronary artery disease. Although abdominal visceral adipose tissue (VAT) plays fundamental roles in coronary artery disease, the relationship between abdominal VAT and AVC is not fully understood. Methods . We investigated 259 patients who underwent cardiac and abdominal computed tomography (CT). AVC was defined as calcified lesion on the aortic valve by CT. %abdominal VAT was calculated as abdominal VAT area/total adipose tissue area. Results . AVC was detected in 75 patients, and these patients showed higher %abdominal VAT (44% versus 38%, p < 0.05 ) compared to those without AVC. When the cutoff value of %abdominal VAT was set at 40.9%, the area under the curve to diagnose AVC was 0.626. Multivariable logistic regression analysis showed that age (OR 1.120, 95% CI 1.078–1.168, p < 0.01 ), diabetes (OR 2.587, 95% CI 1.323–5.130, p < 0.01 ), and %abdominal VAT (OR 1.032, 95% CI 1.003–1.065, p < 0.05 ) were independent risk factors for AVC. The net reclassification improvement value for detecting AVC was increased when %abdominal VAT was added to the model: 0.5093 (95% CI 0.2489–0.7697, p < 0.01 ). Conclusion . We determined that predominance of VAT is associated with AVC.
    Full-text · Article · Jan 2016
    • "To compare the predictive values of the different risk scores on cognition, we used the area under the receiver operating characteristic curves (AUC) [30]. AUC comparisons were fit using the SAS % add_predictive macro. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background: Research concerning the link between individual vascular risk factors and cognition is plentiful but few studies have investigated the role of global vascular risk. We examined the cross-time associations of several vascular risk scores with cognitive performance during aging. Methods: Using data from the French SU.VI.MAX cohort, we studied a sample of 3061 participants. Framingham coronary heart disease, cardiovascular and stroke risk profiles were computed using baseline data (1994-1996). Cognitive performance was assessed after a mean of 13 years via a battery of six validated instruments. Principal component analysis identified scores for verbal memory and working memory. Associations between risk profiles (as continuous variables and in quartiles (Q)) and subsequent poor performance (defined as cognitive score ≤10th percentile) were examined via logistic regression (odds ratios, 95% CI) and analysis of covariance. Results: All continuous-scale Framingham risk scores assessed at midlife were inversely and uniformly associated with subsequent poor global cognitive performance, and especially in terms of verbal memory. Considering risk score Q, higher Q were associated with poorer performance in verbal memory: The fully-adjusted odds ratios (95% CI), comparing Q4 versus Q1, were 2.84 (1.70, 4.75), 2.31 (1.43, 3.73) and 1.77 (1.13, 2.76) for Framingham coronary heart disease, cardiovascular and stroke risk profiles, respectively. Similar findings were observed when modeling cognitive outcomes as continuous variables using covariance analyses. Conclusion: This study supports the existence of an inverse cross-time association between midlife vascular risk profiles and subsequent poor cognitive performance, especially in the verbal memory domain. Beyond their importance as regards vascular risk, such risk scores may help primary prevention efforts in identifying and targeting middle-aged individuals at high risk of cognitive aging.
    No preview · Article · Sep 2015 · Atherosclerosis
  • Source
    • "The net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) were calculated using the software R (Ver. 2.12.1) to examine if the Arg/ADMA ratio may improve the prediction of pathological increase of IMT, compared to the basal model as well as the model including ADMA [27] "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background: Asymmetric dimethylarginine (ADMA), which acts an endogenous inhibitor of nitric oxide synthase (NOS), is involved in the pathogenesis of cardiovascular disease. Arginine (Arg) may regulate vascular endothelial function, since Arg is the substrate of NO competing with ADMA. In our previous study, low Arg/ADMA ratio is an independent risk for microangiopathy-related cerebral damage. Purpose: Here, we performed a cross-sectional study to evaluate the association between the Arg/ADMA ratio and the maximal intima-media thickness (IMT) in the carotid artery. Subjects and Methods: Participants were 785 community-dwelling Japanese people without any severe disorders. Plasma concentration of Arg and ADMA in fasting blood sample was determined using HPLC. IMT was measured in the bilateral carotid artery by ultrasonography. Results: Among quartiles stratified by the Arg/ADMA ratio, ANOVA showed a significant difference in IMT and the IMT in Q1 (the lowest quartile) was significantly higher than that in Q4 (the highest quartile). In multiple linear regression analysis, age, the male gender, lower BMI, the presence of hypertension and lower Arg/ADMA ratio were independently correlated with IMT, while IMT was not correlated with Arg or ADMA alone. In addition, the Arg/ADMA ratio was associated with IMT independent of age, sex, BMI and the presence of hypertension with odds ratio 0.21 (95%CI: 0.05–0.88) in multiple logistic regression analysis for IMT 1.5 mm or more. Conclusion: Imbalance of Arg and ADMA is independently involved in the progression of atherosclerosis, and the Arg/ADMA ratio may be a sensitive marker for atherosclerosis.
    Full-text · Article · Mar 2015 · Atherosclerosis
Show more