Article

Two Criteria for Evaluating Risk Prediction Models

Biostatistics Branch, National Cancer Institute, 6120 Executive Boulevard, Bethesda, Maryland 20892, USA.
Biometrics (Impact Factor: 1.52). 12/2010; 67(3):1057-65. DOI: 10.1111/j.1541-0420.2010.01523.x
Source: PubMed

ABSTRACT We propose and study two criteria to assess the usefulness of models that predict risk of disease incidence for screening and prevention, or the usefulness of prognostic models for management following disease diagnosis. The first criterion, the proportion of cases followed PCF (q), is the proportion of individuals who will develop disease who are included in the proportion q of individuals in the population at highest risk. The second criterion is the proportion needed to follow-up, PNF (p), namely the proportion of the general population at highest risk that one needs to follow in order that a proportion p of those destined to become cases will be followed. PCF (q) assesses the effectiveness of a program that follows 100q% of the population at highest risk. PNF (p) assess the feasibility of covering 100p% of cases by indicating how much of the population at highest risk must be followed. We show the relationship of those two criteria to the Lorenz curve and its inverse, and present distribution theory for estimates of PCF and PNF. We develop new methods, based on influence functions, for inference for a single risk model, and also for comparing the PCFs and PNFs of two risk models, both of which were evaluated in the same validation data.

Download full-text

Full-text

Available from: Ruth Pfeiffer, Dec 22, 2014
1 Follower
 · 
88 Views
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: To estimate the likely number and predictive strength of cancer-associated single nucleotide polymorphisms (SNPs) that are yet to be discovered for seven common cancers. From the statistical power of published genome-wide association studies, we estimated the number of undetected susceptibility loci and the distribution of effect sizes for all cancers. Assuming a log-normal model for risks and multiplicative relative risks for SNPs, family history (FH), and known risk factors, we estimated the area under the receiver operating characteristic curve (AUC) and the proportion of patients with risks above risk thresholds for screening. From additional prevalence data, we estimated the positive predictive value and the ratio of non-patient cases to patient cases (false-positive ratio) for various risk thresholds. Age-specific discriminatory accuracy (AUC) for models including FH and foreseeable SNPs ranged from 0.575 for ovarian cancer to 0.694 for prostate cancer. The proportions of patients in the highest decile of population risk ranged from 16.2% for ovarian cancer to 29.4% for prostate cancer. The corresponding false-positive ratios were 241 for colorectal cancer, 610 for ovarian cancer, and 138 or 280 for breast cancer in women age 50 to 54 or 40 to 44 years, respectively. Foreseeable common SNP discoveries may not permit identification of small subsets of patients that contain most cancers. Usefulness of screening could be diminished by many false positives. Additional strong risk factors are needed to improve risk discrimination.
    Journal of Clinical Oncology 05/2012; 30(17):2157-62. DOI:10.1200/JCO.2011.40.1943 · 17.88 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: An ideal preventive intervention would have negligible side effects and could be applied to the entire population, thus achieving maximal preventive impact. Unfortunately, many interventions have adverse effects and beneficial effects. For example, tamoxifen reduces the risk of breast cancer by about 50% and the risk of hip fracture by 45%, but increases the risk of stroke by about 60%; other serious adverse effects include endometrial cancer and pulmonary embolus. Hence, tamoxifen should only be given to the subset of the population with high enough risks of breast cancer and hip fracture such that the preventive benefits outweigh the risks. Recommendations for preventive use of tamoxifen have been based primarily on breast cancer risk. Age-specific and race-specific rates were considered for other health outcomes, but not risk models. In this paper, we investigate the extent to which modeling not only the risk of breast cancer, but also the risk of stroke, can improve the decision to take tamoxifen. These calculations also give insight into the relative benefits of improving the discriminatory accuracy of such risk models versus improving the preventive effectiveness or reducing the adverse risks of the intervention. Depending on the discriminatory accuracies of the risk models, there may be considerable advantage to modeling the risks of more than one health outcome. Published 2012. This article is a US Government work and is in the public domain in the USA.
    Statistics in Medicine 10/2012; 31(23):2687-96. DOI:10.1002/sim.5443 · 2.04 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: We recently proposed two novel criteria to assess the usefulness of risk prediction models for public health applications. The proportion of cases followed, PCF(p), is the proportion of individuals who will develop disease who are included in the proportion p of individuals in the population at highest risk. The proportion needed to follow-up, PNF(q), is the proportion of the general population at highest risk that one needs to follow in order that a proportion q of those destined to become cases will be followed (Pfeiffer, R. M. and Gail, M. H., 2011. Two criteria for evaluating risk prediction models. Biometrics 67, 1057-1065). Here, we extend these criteria in two ways. First, we introduce two new criteria by integrating PCF and PNF over a range of values of q or p to obtain iPCF, the integrated PCF, and iPNF, the integrated PNF. A key assumption in the previous work was that the risk model is well calibrated. This assumption also underlies novel estimates of iPCF and iPNF based on observed risks in a population alone. The second extension is to propose and study estimates of PCF, PNF, iPCF, and iPNF that are consistent even if the risk models are not well calibrated. These new estimates are obtained from case-control data when the outcome prevalence in the population is known, and from cohort data, with baseline covariates and observed health outcomes. We study the efficiency of the various estimates and propose and compare tests for comparing two risk models, both of which were evaluated in the same validation data.
    Biostatistics 10/2012; 14(2). DOI:10.1093/biostatistics/kxs037 · 2.24 Impact Factor