Ahmed Umar Otokiti’s research while affiliated with White Plains Hospital and other places

BACKGROUND The ubiquity of clinical artificial intelligence (AI) and machine learning (ML) models necessitates measures to ensure the reliability of model output over time. Previous reviews have highlighted the lack of external validation for most clinical models, but a comprehensive review assessing the current priority given to clinical model updating is lacking. OBJECTIVE The objective of this study was to analyze studies of clinical AI models based on PRISMA guidelines. Additionally, a new simple checklist/score system was developed to screen the quality of published AI/ML models. The primary aim was to understand the extent to which clinical model updating is prioritized in current research. METHODS We conducted a systematic analysis of studies on clinical AI models, adhering to PRISMA guidelines. A new checklist/score was introduced to assess the quality of the models. Demographic composition based on ethnicity or race was also considered in the analysis. This comprehensive approach aimed to provide a thorough evaluation of the current landscape of clinical AI models. RESULTS The results of our analysis revealed that only 9% of the reviewed 390 AI/ML studies stated an intention or method to update their models in the future. 98% of the AI/ML models in our review were in the research phase, and only 2 % in production phase. Furthermore, a mere 12% reported following best practice standards for model development. Notably, 84% of the studies did not provide demographic composition based on ethnicity or race. CONCLUSIONS In conclusion, a significant portion of studies currently lack commitment to future model updates. The low adherence to best practice standards for model development also highlights areas for improvement in the field. Furthermore, the absence of demographic information in a substantial number of studies raises concerns about the generalizability and equitable application of these models. The prevalence of research phase models built on proprietary data, limiting independent verification and validation of model output is also a big concern for patient safety. Addressing these issues is crucial for advancing the reliability and inclusivity of clinical AI and ML applications. CLINICALTRIAL NA INTERNATIONAL REGISTERED REPORT RR2-10.2196/37685

BACKGROUND With an increase in the number of artificial intelligence (AI) and machine learning (ML) algorithms available for clinical settings, appropriate model updating and implementation of updates are imperative to ensure applicability, reproducibility, and patient safety. OBJECTIVE The objective of this scoping review was to evaluate and assess the model-updating practices of AI and ML clinical models that are used in direct patient-provider clinical decision-making. METHODS We used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist and the PRISMA-P protocol guidance in addition to a modified CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) checklist to conduct this scoping review. A comprehensive medical literature search of databases, including Embase, MEDLINE, PsycINFO, Cochrane, Scopus, and Web of Science, was conducted to identify AI and ML algorithms that would impact clinical decision-making at the level of direct patient care. Our primary end point is the rate at which model updating is recommended by published algorithms; we will also conduct an assessment of study quality and risk of bias in all publications reviewed. In addition, we will evaluate the rate at which published algorithms include ethnic and gender demographic distribution information in their training data as a secondary end point. RESULTS Our initial literature search yielded approximately 13,693 articles, with approximately 7810 articles to consider for full reviews among our team of 7 reviewers. We plan to complete the review process and disseminate the results by spring of 2023. CONCLUSIONS Although AI and ML applications in health care have the potential to improve patient care by reducing errors between measurement and model output, currently there exists more hype than hope because of the lack of proper external validation of these models. We expect to find that the AI and ML model-updating methods are proxies for model applicability and generalizability on implementation. Our findings will add to the field by determining the degree to which published models meet the criteria for clinical validity, real-life implementation, and best practices to optimize model development, and in so doing, reduce the overpromise and underachievement of the contemporary model development process. INTERNATIONAL REGISTERED REPORT PRR1-10.2196/37685

Background With an increase in the number of artificial intelligence (AI) and machine learning (ML) algorithms available for clinical settings, appropriate model updating and implementation of updates are imperative to ensure applicability, reproducibility, and patient safety. Objective The objective of this scoping review was to evaluate and assess the model-updating practices of AI and ML clinical models that are used in direct patient-provider clinical decision-making. Methods We used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist and the PRISMA-P protocol guidance in addition to a modified CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) checklist to conduct this scoping review. A comprehensive medical literature search of databases, including Embase, MEDLINE, PsycINFO, Cochrane, Scopus, and Web of Science, was conducted to identify AI and ML algorithms that would impact clinical decision-making at the level of direct patient care. Our primary end point is the rate at which model updating is recommended by published algorithms; we will also conduct an assessment of study quality and risk of bias in all publications reviewed. In addition, we will evaluate the rate at which published algorithms include ethnic and gender demographic distribution information in their training data as a secondary end point. Results Our initial literature search yielded approximately 13,693 articles, with approximately 7810 articles to consider for full reviews among our team of 7 reviewers. We plan to complete the review process and disseminate the results by spring of 2023. Conclusions Although AI and ML applications in health care have the potential to improve patient care by reducing errors between measurement and model output, currently there exists more hype than hope because of the lack of proper external validation of these models. We expect to find that the AI and ML model-updating methods are proxies for model applicability and generalizability on implementation. Our findings will add to the field by determining the degree to which published models meet the criteria for clinical validity, real-life implementation, and best practices to optimize model development, and in so doing, reduce the overpromise and underachievement of the contemporary model development process. International Registered Report Identifier (IRRID) PRR1-10.2196/37685