Karel G M Moons’s research while affiliated with University Medical Center Utrecht and other places

Background: This paper details initial testing of the agreeability and usability of a novel quality appraisal tool for systematic reviews of prognostic factor studies: AMSTAR-PF. Methods: Fourteen appraisers each assessed eight systematic reviews using AMSTAR-PF. Their ratings for each question and each article were compared, with interrater, inter-pair and intrapair agreeability calculated using Gwet's agreement coefficient. Time of use and time to reach consensus were also recorded. Results: Interrater agreement averaged 0.59 (range, 0.21-0.90), inter-pair 0.61 (range 0.24-0.91) and intrapair 0.75 (range 0.45-0.95) across the domains, with agreement for the overall rating 0.46 (95%CI 0.30-0.62) for interrater, 0.46 (95%CI 0.17-0.74) for inter-pair, and 0.68 (range of averages 0.22-1.00) for intrapair agreement. The majority (60.7%) of intrapair ratings were identical, with 94.6% of final ratings either identical or only one category different for the overall appraisal. The time taken to appraise a study with AMSTAR-PF improved with use and averaged around 34 minutes after the first two appraisals. Conclusions: Despite some variance in agreeability for different domains and between different appraisers, the testing results suggest that AMSTAR-PF has clear utility for appraising the quality of systematic reviews of prognostic factor studies.

The Prediction model Risk Of Bias ASsessment Tool (PROBAST) is used to assess the quality, risk of bias, and applicability of prediction models or algorithms and of prediction model/algorithm studies. Since PROBAST’s introduction in 2019, much progress has been made in the methodology for prediction modelling and in the use of artificial intelligence, including machine learning, techniques. An update to PROBAST-2019 is thus needed. This article describes the development of PROBAST+AI. PROBAST+AI consists of two distinctive parts: model development and model evaluation. For model development, PROBAST+AI users assess quality and applicability using 16 targeted signalling questions. For model evaluation, PROBAST+AI users assess the risk of bias and applicability using 18 targeted signalling questions. Both parts contain four domains: participants and data sources, predictors, outcome, and analysis. Applicability of the prediction model is rated for the participants and data sources, predictors, and outcome domains. PROBAST+AI may replace the original PROBAST tool and allows all key stakeholders (eg, model developers, AI companies, researchers, editors, reviewers, healthcare professionals, guideline developers, and policy organisations) to examine the quality, risk of bias, and applicability of any type of prediction model in the healthcare sector, irrespective of whether regression modelling or AI techniques are used.

The disease course and outcome of COVID-19 greatly varies between individuals. To explore which biological systems may contribute to this variation, we examined how individual metabolites and three metabolic scores relate to COVID-19 outcomes in hospitalized COVID-19 patients. The metabolome of 346 patients was measured using the 1H-NMR Nightingale platform. The association of individual metabolomic features and multi-biomarker scores, i.e. MetaboHealth, MetaboAge, and Infectious Disease Score (IDS) (higher scores reflect poorer health), with in-hospital disease course, long-term recovery, and overall survival were analyzed. Higher values for the metabolites phenylalanine (HR = 1.33, CI = 1.14–1.56), glucose (HR = 1.37, CI = 1.16–1.62) and lactate (HR = 1.38, CI = 1.16–1.63) were associated with mortality. For all three metabolic scores, higher scores were significantly associated with higher odds of a poorer in-hospital disease course (MetaboHealth: OR = 1.61, CI = 1.29–2.02; ΔMetaboAge: OR = 1.42, CI = 1.16–1.74; IDS: OR = 1.55, 1.25–1.93) and with overall survival (MetaboHealth: HR = 1.57, CI = 1.28–1.92; ΔMetaboAge: HR = 1.34, CI = 1.15–1.57; IDS: HR = 1.56, CI = 1.27–1.93). MetaboHealth and ΔMetaboAge showed a stronger association in younger patients (< 70 yrs.) than older patients. No clear patterns were found in associations between the three scores and measures of long-term recovery. In conclusion, the heterogeneity in disease course after SARS-COV2 infection may be explained either by generic biological frailty reflected by the three metabolomics scores or by glycemic control (glucose, lactate) and respiratory distress (phenylalanine).

Introduction Risk prediction models are increasingly used in healthcare to aid in clinical decision‐making. In most clinical contexts, model calibration (i.e., assessing the reliability of risk estimates) is critical. Data available for model development are often not perfectly balanced with the modeled outcome (i.e., individuals with vs. without the event of interest are not equally prevalent in the data). It is common for researchers to correct for class imbalance, yet, the effect of such imbalance corrections on the calibration of machine learning models is largely unknown. Methods We studied the effect of imbalance corrections on model calibration for a variety of machine learning algorithms. Using extensive Monte Carlo simulations we compared the out‐of‐sample predictive performance of models developed with an imbalance correction to those developed without a correction for class imbalance across different data‐generating scenarios (varying sample size, the number of predictors, and event fraction). Our findings were illustrated in a case study using MIMIC‐III data. Results In all simulation scenarios, prediction models developed without a correction for class imbalance consistently had equal or better calibration performance than prediction models developed with a correction for class imbalance. The miscalibration introduced by correcting for class imbalance was characterized by an over‐estimation of risk and was not always able to be corrected with re‐calibration. Conclusion Correcting for class imbalance is not always necessary and may even be harmful to clinical prediction models which aim to produce reliable risk estimates on an individual basis.

Objectives The Wells rule is often used in primary care to rule out pulmonary embolism (PE), but its efficiency is low as many referred patients do not have PE. In this study, we evaluated in primary care an alternative and potentially more efficient diagnostic strategy—the YEARS algorithm; a simplified three-item version of the Wells rule combined with a pretest probability adjusted D-dimer interpretation. Design In this comprehensive prospective diagnostic validation study, primary care patients suspected of PE were enrolled by their general practitioner. All three YEARS items were collected in addition to D-dimer results, and patients were followed for 3 months to establish the final diagnosis. Setting Primary care in the Netherlands. Participants 753 patients with suspected acute PE were included. Five patients (0.7%) were lost to follow-up. Main outcome measures Failure rate (number of PE cases among patients classified by the algorithm as ‘PE ruled-out’) and efficiency (fraction of patients classified as ‘PE probable/further imaging needed’). Results Prevalence of PE was 5.5% (41/748 patients). In total, 603 patients were classified as ‘PE ruled-out’ by the YEARS algorithm (532 with zero YEARS items and a D-dimer<1000 ng/mL and 71 with≥1 positive YEARS item and a D-dimer<500 ng/mL), resulting in an efficiency of 80.6% (603/748 patients, 95% CI 77.6% to 83.4%). Of these patients, three patients had a diagnosis of non-fatal PE during 3 months follow-up, all three with zero YEARS items and D-dimer between 500 and 1000 ng/mL, resulting in an overall diagnostic failure rate of 0.50% (3/603 patients, 95% CI 0.13% to 1.57%). In the patients categorised as ‘imaging needed’ (n=145), a total of 38 (26.2%) were indeed diagnosed with PE. Conclusions Our study suggests that acute PE can be safely ruled out in 80% of patients by the YEARS algorithm in a primary care setting, while only 20% of patients required referral to hospital care for imaging tests. In those classified as ‘imaging needed’, PE was present in about one in every four patients, demonstrating a high detection proportion.

Background During the COVID-19 pandemic, nursing home (NH) residents faced the highest risk of severe COVID-19 disease and mortality. Due to their frailty status, comorbidity burden can serve as a useful predictive indicator of vulnerability in this frail population. However, the prognostic value of these cumulative comorbidity scores like the Charlson comorbidity index (CCI) remained unclear in this population. We evaluated the incremental predictive value of the CCI for predicting 28-day mortality in NH residents with COVID-19, compared to prediction using age and sex only. Methods We included older individuals of ≥ 70 years of age in a large retrospective observational cohort across NHs in the Netherlands. Individuals with PCR-confirmed COVID-19 diagnosis from 1 March 2020 to 31 December 2021 were included. The CCI score was computed by searching for the comorbidities recorded in the electronic patient records. All-cause mortality within 28 days was predicted using logistic regression based on age and sex only (base model) and by adding the CCI to the base model (CCI model). The predictive performance of the base model and the CCI model were compared visually by the distribution of predicted risks and area under the receiver operator characteristic curve (AUROC), scaled Brier score, and calibration slope. Results A total of 4318 older NH residents were included in this study with a median age of 88 years [IQR: 83–93] and a median CCI score of 6 [IQR: 5–7]. 1357 (31%) residents died within 28 days after COVID-19 diagnosis. The base model, with age and sex as predictors, had an AUROC of 0.61 (CI: 0.60 to 0.63), a scaled brier score of 0.03 (CI: 0.02 to 0.04), and a calibration slope of 0.97 (CI: 0.83 to 1.13). The addition of CCI did not improve these predictive performance measures. Conclusion The addition of the CCI as a vulnerability indicator did not improve short-term mortality prediction in NH residents. Similar (high) age and number of comorbidities in the NH population could reduce the effectiveness of these predictors, emphasizing the need for other population-specific predictors that can be utilized in the frail NH residents. Graphical Abstract

Large language models (LLMs) are rapidly being adopted in healthcare, necessitating standardized reporting guidelines. We present transparent reporting of a multivariable model for individual prognosis or diagnosis (TRIPOD)-LLM, an extension of the TRIPOD + artificial intelligence statement, addressing the unique challenges of LLMs in biomedical applications. TRIPOD-LLM provides a comprehensive checklist of 19 main items and 50 subitems, covering key aspects from title to discussion. The guidelines introduce a modular format accommodating various LLM research designs and tasks, with 14 main items and 32 subitems applicable across all categories. Developed through an expedited Delphi process and expert consensus, TRIPOD-LLM emphasizes transparency, human oversight and task-specific performance reporting. We also introduce an interactive website ( https://tripod-llm.vercel.app/ ) facilitating easy guideline completion and PDF generation for submission. As a living document, TRIPOD-LLM will evolve with the field, aiming to enhance the quality, reproducibility and clinical applicability of LLM research in healthcare through comprehensive reporting.

Background We evaluated the performance of prognostic models for predicting mortality or ICU admission in hospitalized patients with COVID-19 in the World Health Organization (WHO) Global Clinical Platform, a repository of individual-level clinical data of patients hospitalized with COVID-19, including in low- and middle-income countries (LMICs). Methods We identified eligible multivariable prognostic models for predicting overall mortality and ICU admission during hospital stay in patients with confirmed or suspected COVID-19 from a living review of COVID-19 prediction models. These models were evaluated using data contributed to the WHO Global Clinical Platform for COVID-19 from nine LMICs (Burkina Faso, Cameroon, Democratic Republic of Congo, Guinea, India, Niger, Nigeria, Zambia, and Zimbabwe). Model performance was assessed in terms of discrimination and calibration. Results Out of 144 eligible models, 140 were excluded due to a high risk of bias, predictors unavailable in LIMCs, or insufficient model description. Among 11,338 participants, the remaining models showed good discrimination for predicting in-hospital mortality (3 models), with areas under the curve (AUCs) ranging between 0.76 (95% CI 0.71–0.81) and 0.84 (95% CI 0.77–0.89). An AUC of 0.74 (95% CI 0.70–0.78) was found for predicting ICU admission risk (one model). All models showed signs of miscalibration and overfitting, with extensive heterogeneity between countries. Conclusions Among the available COVID-19 prognostic models, only a few could be validated on data collected from LMICs, mainly due to limited predictor availability. Despite their discriminative ability, selected models for mortality prediction or ICU admission showed varying and suboptimal calibration.

A myriad of measures to illustrate performance of predictive artificial intelligence (AI) models have been proposed in the literature. Selecting appropriate performance measures is essential for predictive AI models that are developed to be used in medical practice, because poorly performing models may harm patients and lead to increased costs. We aim to assess the merits of classic and contemporary performance measures when validating predictive AI models for use in medical practice. We focus on models with a binary outcome. We discuss 32 performance measures covering five performance domains (discrimination, calibration, overall, classification, and clinical utility) along with accompanying graphical assessments. The first four domains cover statistical performance, the fifth domain covers decision-analytic performance. We explain why two key characteristics are important when selecting which performance measures to assess: (1) whether the measure's expected value is optimized when it is calculated using the correct probabilities (i.e., a "proper" measure), and (2) whether they reflect either purely statistical performance or decision-analytic performance by properly considering misclassification costs. Seventeen measures exhibit both characteristics, fourteen measures exhibited one characteristic, and one measure possessed neither characteristic (the F1 measure). All classification measures (such as classification accuracy and F1) are improper for clinically relevant decision thresholds other than 0.5 or the prevalence. We recommend the following measures and plots as essential to report: AUROC, calibration plot, a clinical utility measure such as net benefit with decision curve analysis, and a plot with probability distributions per outcome category.