Ben Van Calster

• PhD
• Professor at KU Leuven

Evaluation of clinical prediction models across multiple clusters, whether centers or datasets, is becoming increasingly common. A comprehensive evaluation includes an assessment of the agreement between the estimated risks and the observed outcomes, also known as calibration. Calibration is of utmost importance for clinical decision making with pr...

Introduction Risk prediction models are increasingly used in healthcare to aid in clinical decision‐making. In most clinical contexts, model calibration (i.e., assessing the reliability of risk estimates) is critical. Data available for model development are often not perfectly balanced with the modeled outcome (i.e., individuals with vs. without t...

Background Non-typhoidal Salmonella (NTS) frequently cause bloodstream infection in children under-five in sub-Saharan Africa, particularly in malaria-endemic areas. Due to increasing drug resistance, NTS are often not covered by standard-of-care empirical antibiotics for severe febrile illness. We developed a clinical prediction model to orient th...

Dynamic predictive modeling using electronic health record (EHR) data has gained significant attention in recent years. The reliability and trustworthiness of such models depend heavily on the quality of the underlying data, which is largely determined by the stages preceding the model development: data extraction from EHR systems and data preparat...

Objective: To prospectively validate the performance of the Risk of Malignancy Index (RMI), International Ovarian Tumor Analysis (IOTA) Simple Rules Risk Model (SRRisk), IOTA Assessment of Different NEoplasias in the adneXa (ADNEX) and the IOTA two-step strategy in different types of ultrasound centers in Italy. Methods: This is a multicenter prosp...

Objectives Identifying cardiac surgical patients at risk of requiring red blood cell (RBC) transfusion is crucial for optimizing their outcome. We critically appraised prognostic models preoperatively predicting perioperative exposure to RBC transfusion in adult cardiac surgery and summarized model performance. Methods Design : Systematic review a...

A myriad of measures to illustrate performance of predictive artificial intelligence (AI) models have been proposed in the literature. Selecting appropriate performance measures is essential for predictive AI models that are developed to be used in medical practice, because poorly performing models may harm patients and lead to increased costs. We...

Background: ADNEX and RMI are models to estimate the risk of malignancy of ovarian masses based on clinical and ultrasound information. The aim of this systematic review and meta-analysis is to synthesise head to-head comparisons of these models. Methods: We performed a systematic literature search up to 31/07/2024. We included all external validat...

Background: Central line-associated bloodstream infections (CLABSI) are preventable hospital-acquired infections. Predicting CLABSI helps improve early intervention strategies and enhance patient safety. Aim: To develop and temporally evaluate dynamic prediction models for continuous CLABSI risk monitoring. Methods: Data from hospitalized patients...

Introduction ROCkeTS investigated accuracy of risk prediction models; no previous studies investigate all tests as head-to-head comparisons. Methods Study design – prospective cohort study.Recruitment – newly presenting premenopausal women with non-specific symptoms and raised CA125 and/or abnormal imaging underwent Risk of malignancy algorithm (R...

Background Random forests have become popular for clinical risk prediction modeling. In a case study on predicting ovarian malignancy, we observed training AUCs close to 1. Although this suggests overfitting, performance was competitive on test data. We aimed to understand the behavior of random forests for probability estimation by (1) visualizing...

Prediction models are used to predict an outcome based on input variables. Missing data in input variables often occurs at model development and at prediction time. The missForestPredict R package proposes an adaptation of the missForest imputation algorithm that is fast, user-friendly and tailored for prediction settings. The algorithm iteratively...

Background We previously proposed two cfDNA-based scores (genome-wide z-score and nucleosome score) as candidate non-invasive biomarkers to further improve pre-surgical diagnosis of ovarian malignancy. We aimed to investigate the added value of these cfDNA-based scores to the predictors of the ADNEX model (Assessment of Different NEoplasias in the...

Objective: Prognostic outcomes related to hospital admissions typically do not suffer from censoring, and can be modeled either categorically or as time-to-event. Competing events are common but often ignored. We compared the performance of random forest (RF) models to predict the risk of central line-associated bloodstream infections (CLABSI) usin...

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement was published in 2015 to provide the minimum reporting recommendations for studies developing or evaluating the performance of a prediction model. Methodological advances in the field of prediction have since included the widesprea...

Background The ADNEX model (Assessment of Different NEoplasias in the adnexa) is the best performing model to predict the risk of malignancy (binary) and type of malignancy (multiclass) in ovarian tumors. The immune system plays a role in the onset and progression of ovarian cancer. Preliminary research has suggested that immune-related biomarkers...

Objectives The aim is to evaluate the ability of the Assessment of Different NEoplasias in the adneXa model (ADNEX) and the International Ovarian Tumour Analysis (IOTA) two‐step strategy to predict malignancy in adnexal masses detected in an outpatient low‐risk setting, and to estimate the risk of complications in masses with benign ultrasound morp...

Random forests have become popular for clinical risk prediction modelling. In a case study on predicting ovarian malignancy, we observed training c-statistics close to 1. Although this suggests overfitting, performance was competitive on test data. We aimed to understand the behaviour of random forests by (1) visualizing data space in three real wo...

Objectives To conduct a systematic review of studies externally validating the ADNEX (Assessment of Different Neoplasias in the adnexa) model for diagnosis of ovarian cancer and to present a meta-analysis of its performance. Design Systematic review and meta-analysis of external validation studies Data sources Medline, Embase, Web of Science, Sco...

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, w...

Background: Several diagnostic prediction models to help clinicians discriminate between benign and malignant adnexal masses are available. This study is a head-to-head comparison of the performance of the Assessment of Different NEoplasias in the adneXa (ADNEX) model with that of the Risk of Ovarian Malignancy Algorithm (ROMA). Methods: This is...

Tuning hyperparameters, such as the regularization parameter in Ridge or Lasso regression, is often aimed at improving the predictive performance of risk prediction models. In this study, various hyperparameter tuning procedures for clinical prediction models were systematically compared and evaluated in low‐dimensional data. The focus was on out‐o...

Evaluating the performance of a clinical prediction model is crucial to establish its predictive accuracy in the populations and settings intended for use. In this article, the first in a three part series, Collins and colleagues describe the importance of a meaningful evaluation using internal, internal-external, and external validation, as well a...

Background Assessing malignancy risk is important to choose appropriate management of ovarian tumors. We compared six algorithms to estimate the probabilities that an ovarian tumor is benign, borderline malignant, stage I primary invasive, stage II-IV primary invasive, or secondary metastatic. Methods This retrospective cohort study used 5909 pati...

Introduction Ectopic pregnancy (EP) is the major high‐risk outcome following a pregnancy of unknown location (PUL) classification. Biochemical markers are used to triage PUL to high versus low risk to decide appropriate follow up. The M6 model is currently the best risk prediction model. We aimed to update the M6 model and evaluate whether the mode...

Objective: To systematically review the risk of bias and applicability of published prediction models for risk of central line-associated bloodstream infection (CLA-BSI) in hospitalized patients. Study design and setting: Systematic review of literature in PubMed, Embase, Web of Science Core Collection and Scopus up to July 10, 2023. Two authors...

OBJECTIVE. To compare performance and probability estimates of six algorithms to estimate the probabilities that an ovarian tumor is benign, borderline malignant, stage I primary invasive, stage II-IV primary invasive, or secondary metastatic. MATERIALS AND METHODS. Models were developed on 5909 patients (recruited 1999-2012) and validated on 3199...

Objectives: To conduct a systematic review of studies externally validating the ADNEX model for ovarian cancer diagnosis and perform a meta-analysis of its performance. Design: Systematic review, meta-analysis Data sources: Medline, EMBASE, WOS, Scopus, and EuropePMC up to 15/05/2023. Review methods: We included external validation studies of the p...

Background There are numerous methods available to develop clinical prediction models to estimate the risks of a nominal polytomous outcome. A comprehensive evaluation of the most appropriate method has not yet been undertaken. We compared the predictive performance of a range of models in a simulation study and illustrate how to implement them wit...

Background Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context? Main body We argue to the contrary because (1) patient populations vary, (2) me...

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem....

The TRIPOD-Cluster (transparent reporting of multivariable prediction models developed or validated using clustered data) statement comprises a 19 item checklist, which aims to improve the reporting of studies developing or validating a prediction model in clustered data, such as individual participant data meta-analyses (clustering by study) and e...

The increasing availability of large combined datasets (or big data), such as those from electronic health records and from individual participant data meta-analyses, provides new opportunities and challenges for researchers developing and validating (including updating) prediction models. These datasets typically include individuals from multiple...

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem....

Aims Multinomial logistic regression models allow one to predict the risk of a categorical outcome with > 2 categories. When developing such a model, researchers should ensure the number of participants ( n ) is appropriate relative to the number of events ( E k ) and the number of predictor parameters ( p k ) for each category k. We propose three...

Objectives: To systematically review the risk of bias and applicability of published prediction models for risk of central line-associated bloodstream infection (CLA-BSI) in hospitalized patients. Introduction: CLA-BSIs are the most common source of hospital-acquired infections (HAIs), and are always associated with higher morbidity, longer length...

Objectives: To systematically review the risk of bias and applicability of published prediction models for risk of central line-associated bloodstream infection (CLA-BSI) in hospitalized patients. Introduction: CLA-BSIs are the most common source of hospital-acquired infections (HAIs), and are always associated with higher morbidity, longer length...

Objectives: To systematically review the risk of bias and applicability of published prediction models for risk of central line-associated bloodstream infection (CLA-BSI) in hospitalized patients. Introduction: CLA-BSIs are the most common source of hospital-acquired infections (HAIs), and are always associated with higher morbidity, longer length...

Objectives: To systematically review the risk of bias and applicability of published prediction models for risk of central line-associated bloodstream infection (CLA-BSI) in hospitalized patients. Introduction: CLA-BSIs are the most common source of hospital-acquired infections (HAIs), and are always associated with higher morbidity, longer length...

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from sur...

Importance Correct diagnosis of ovarian cancer results in better prognosis. Adnexal lesions can be stratified into the Ovarian-Adnexal Reporting and Data System (O-RADS) risk of malignancy categories with either the O-RADS lexicon, proposed by the American College of Radiology, or the International Ovarian Tumor Analysis (IOTA) 2-step strategy. Ob...

Objective: To assess improvement in the completeness of reporting COVID-19 prediction models after the peer review process. Study design and setting: Studies included in a living systematic review of COVID-19 prediction models, with both pre-print and peer-reviewed published versions available, were assessed. The primary outcome was the change i...

Objective: Previous work suggested that the ultrasound-based benign Simple Descriptors can reliably exclude malignancy in a large proportion of women presenting with an adnexal mass. We aim to validate a modified version of the Benign Simple Descriptors (BD), and we introduce a two-step strategy to estimate the risk of malignancy: if the BDs do no...

Multinomial logistic regression models allow one to predict the risk of a categorical outcome with more than 2 categories. When developing such a model, researchers should ensure the number of participants (n) is appropriate relative to the number of events (E.k) and the number of predictor parameters (p.k) for each category k. We propose three cri...

Objective To externally validate various prognostic models and scoring rules for predicting short term mortality in patients admitted to hospital for covid-19. Design Two stage individual participant data meta-analysis. Setting Secondary and tertiary care. Participants 46 914 patients across 18 countries, admitted to a hospital with polymerase c...

Background Prognostic models are used widely in the oncology domain to guide medical decision-making. Little is known about the risk of bias of prognostic models developed using machine learning and the barriers to their clinical uptake in the oncology domain. Methods We conducted a systematic review and searched MEDLINE and EMBASE databases for o...

There has been increased interest in the use of clinical risk prediction models for decision-making in medicine for patient care. This has been accelerated through the focus on precision medicine, the revolution in omics data, and increasing use of randomized controlled trial and electronic health record databases. These models are expected to assi...

Objective: Methods to correct class imbalance (imbalance between the frequency of outcome events and nonevents) are receiving increasing interest for developing prediction models. We examined the effect of imbalance correction on the performance of logistic regression models. Material and methods: Prediction models were developed using standard...

The field of automatic biomedical image analysis crucially depends on robust and meaningful performance metrics for algorithm validation. Current metric usage, however, is often ill-informed and does not reflect the underlying domain interest. Here, we present a comprehensive framework that guides researchers towards choosing performance metrics in...

Objective: Assess the prognostic value of pre-operative haemoglobin concentration (Hb) for identifying patients who develop severe post-operative anaemia or require blood transfusion following primary total hip or knee, or unicompartmental knee arthroplasty (THA, TKA, UKA). Background: Pre-operative group and save (G&S), and post-operative Hb me...

Fragmentation patterns of plasma cell-free DNA (cfDNA) are known to reflect nucleosome positions of cell types contributing to cfDNA. Based on cfDNA fragmentation patterns, the deviation in nucleosome footprints was quantified between diagnosed ovarian cancer patients and healthy individuals. Multinomial modeling was subsequently applied to capture...

Calibration is a vital aspect of the performance of risk prediction models, but research in the context of ordinal outcomes is scarce. This study compared calibration measures for risk models predicting a discrete ordinal outcome, and investigated the impact of the proportional odds assumption on calibration and overfitting. We studied the multinom...

supplement https://onlinelibrary.wiley.com/action/downloadSupplement?doi=10.1002%2Fsim.9281&file=sim9281-sup-0001-supinfo.pdf

Background Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology. Methods We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used...

BACKGROUND While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood. METHODS We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary synd...

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. We aim to give a description of measures to evaluate predictions and the potential improvement in decision making...

Objectives To investigate prognostic factors for anxiety, depression and post-traumatic stress (PTS) symptoms 1 month after early pregnancy loss (EPL). Design A prospective cohort study. Consecutive women were recruited, and demographic and clinical data were collected. Surveys containing the Hospital Anxiety and Depression Scale (HADS) and Post-t...

Objectives: To develop a model that can discriminate between different etiologies of abnormal uterine bleeding. Design: The International Endometrial Tumor Analysis (IETA) 1 study is a multicenter observational diagnostic study in 18 bleeding clinics in 9 countries. Consecutive women with abnormal vaginal bleeding presenting for ultrasound exami...

supplement to "Risk prediction models for discrete ordinal outcomes: calibration and the impact of the proportional odds assumption"

Introduction There is no global agreement on how to best determine pregnancy of unknown location viability and location using biomarkers. Measurements of progesterone and β human chorionic gonadotropin (βhCG) are still used in clinical practice to exclude the possibility of a viable intrauterine pregnancy (VIUP). We evaluate the predictive value of...

Background: There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated. Methods: A SCOPUS citation search was run on March 22, 2017 to identify e...

Medical multi-category diagnostic problems may involve discrete biomarkers. Many traditional accuracy measures are based on the assumption that all biomarkers follow continuous distributions and consequently may underestimate the true discrimination ability of the discrete markers. In particular, we focus on Hypervolume Under ROC Manifold (HUM) in...

Introduction The Transparent Reporting of a multivariable prediction model of Individual Prognosis Or Diagnosis (TRIPOD) statement and the Prediction model Risk Of Bias ASsessment Tool (PROBAST) were both published to improve the reporting and critical appraisal of prediction model studies for diagnosis and prognosis. This paper describes the proce...

Objective: Evaluate the completeness of reporting of prognostic prediction models developed using machine learning methods in the field of oncology. Study design and setting: We conducted a systematic review, searching the MEDLINE and Embase databases between 01/01/2019 and 05/09/2019, for non-imaging studies developing a prognostic clinical pre...

Covid-19 research made it painfully clear that the scandal of poor medical research, as denounced by Altman in 1994, persists today. The overall quality of medical research remains poor, despite longstanding criticisms. The problems are well known, but the research community fails to properly address them. We suggest most problems stem from an unde...

Calibration is a vital aspect of the performance of risk prediction models, but research in the context of ordinal outcomes is scarce. This study compared calibration measures for risk models predicting a discrete ordinal outcome, and investigated the impact of the proportional odds assumption on calibration and overfitting. We studied the multinom...