Allison McCoy’s research while affiliated with Vanderbilt University and other places

Objective The objectives of this study are to synthesize findings from recent research of retrieval-augmented generation (RAG) and large language models (LLMs) in biomedicine and provide clinical development guidelines to improve effectiveness. Materials and Methods We conducted a systematic literature review and a meta-analysis. The report was created in adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 analysis. Searches were performed in 3 databases (PubMed, Embase, PsycINFO) using terms related to “retrieval augmented generation” and “large language model,” for articles published in 2023 and 2024. We selected studies that compared baseline LLM performance with RAG performance. We developed a random-effect meta-analysis model, using odds ratio as the effect size. Results Among 335 studies, 20 were included in this literature review. The pooled effect size was 1.35, with a 95% confidence interval of 1.19-1.53, indicating a statistically significant effect (P = .001). We reported clinical tasks, baseline LLMs, retrieval sources and strategies, as well as evaluation methods. Discussion Building on our literature review, we developed Guidelines for Unified Implementation and Development of Enhanced LLM Applications with RAG in Clinical Settings to inform clinical applications using RAG. Conclusion Overall, RAG implementation showed a 1.35 odds ratio increase in performance compared to baseline LLMs. Future research should focus on (1) system-level enhancement: the combination of RAG and agent, (2) knowledge-level enhancement: deep integration of knowledge into LLM, and (3) integration-level enhancement: integrating RAG systems within electronic health records.

Introduction: Patient reported outcomes (PRO) including patient disease activity scores, quality of life, and depressive symptoms are increasingly being used for clinical care in patients with inflammatory bowel disease (IBD). However, little is known about performance of PROs over time in a real-world setting. Methods: PROs were collected from a tertiary care IBD center from the electronic medical records (EMR) from 2018–2020. Quality of life was measured with the Simple Inflammatory Bowel Disease Questionnaire (SIBDQ). Disease activity was measured using the Harvey Bradshaw Index (HBI) for Crohn’s disease (CD) and the Simple Clinical Colitis Activity Index (SCCAI) for ulcerative colitis (UC). Depressive symptoms were measured using the Patient Health Questionnaire-8 (PHQ-8). Results: PRO collection rate from 1,373 patients was 88%. PHQ and SIBDQ were highly correlated (0.718, p < 0.05). HBI and PHQ-8 (0.528, p < 0.05) and SIBDQ (-0.676, p < 0.05) were moderately correlated. For CD patients, age 40–65, corticosteroid use, and psychiatric medication use were associated with lower HBI and SIBDQ and higher PHQ scores. Age 40–65, corticosteroid use, and psychiatric medication use were associated with lower SCCAI and SIBDQ and higher PHQ-8 scores for CD and UC patients. Mean change [95% CI] increased for SIBDQ by 2.9 [2.3 to 3.6] in CD and 4.7 [3.8 to 5.6] in UC. Mean change [95% CI] decreased for PHQ-8: -1.0 [-1.3 to -0.8] in CD and − 1.7 [-2.2 to -1.3] in UC. This remained significant in both cognitive and somatic subscales. Conclusion: PRO collection was highly successful using EMR implementation in a tertiary care clinic setting. Corticosteroid use and psychiatric medication use were associated with worse disease activity, depressive symptom and quality of life scores. Moreover, PROs demonstrated depressive symptoms (both cognitive and somatic subscales), and quality of life symptom scores improved over time.

Clinical prediction models have been widely acknowledged as informative tools providing evidence‐based support for clinical decision making. However, prediction models are often underused in clinical practice due to many reasons including missing information upon real‐time risk calculation in electronic health records (EHR) system. Existing literature to address this challenge focuses on statistical comparison of various approaches while overlooking the feasibility of their implementation in EHR. In this article, we propose a novel and feasible submodel approach to address this challenge for prediction models developed using the model approximation (also termed “preconditioning”) method. The proposed submodel coefficients are equivalent to the corresponding original prediction model coefficients plus a correction factor. Comprehensive simulations were conducted to assess the performance of the proposed method and compared with the existing “one‐step‐sweep” approach as well as the imputation approach. In general, the simulation results show the preconditioning‐based submodel approach is robust to various heterogeneity scenarios and is comparable to the imputation‐based approach, while the “one‐step‐sweep” approach is less robust under certain heterogeneity scenarios. The proposed method was applied to facilitate real‐time implementation of a prediction model to identify emergency department patients with acute heart failure who can be safely discharged home.

Background Over the past 30 years, the American Medical Informatics Association (AMIA) has played a pivotal role in fostering a collaborative community for professionals in biomedical and health informatics. As an interdisciplinary association, AMIA brings together individuals with clinical, research, and computer expertise and emphasizes the use of data to enhance biomedical research and clinical work. The need for a recognition program within AMIA, acknowledging applied informatics skills by members, led to the establishment of the Fellows of AMIA (FAMIA) Recognition Program in 2018. Objectives To outline the evolution of the FAMIA program and shed light on its origins, development, and impact. This report explores factors that led to the establishment of FAMIA, considerations affecting its development, and the objectives FAMIA seeks to achieve within the broader context of AMIA. Methods The development of FAMIA is examined through a historical lens, encompassing key milestones, discussions, and decisions that shaped the program. Insights into the formation of FAMIA were gathered through discussions within AMIA membership and leadership, including proposals, board-level discussions, and the involvement of key stakeholders. Additionally, the report outlines criteria for FAMIA eligibility and the pathways available for recognition, namely the Certification Pathway and the Long-Term Experience Pathway. Results The FAMIA program has inducted five classes, totaling 602 fellows. An overview of disciplines, roles, and application pathways for FAMIA members is provided. A comparative analysis with other fellow recognition programs in related fields showcases the unique features and contributions of FAMIA in acknowledging applied informatics. Conclusion Now in its sixth year, FAMIA acknowledges the growing influence of applied informatics within health information professionals, recognizing individuals with experience, training, and a commitment to the highest level of applied informatics and the science associated with it.

INTRODUCTION Telehealth care in the ambulatory setting has dramatically increased in the COVID and post-COVID period in medicine. Understanding how this may impact the care of patients (pts) with inflammatory bowel disease (IBD) is important. This study aims to evaluate the pattern of use of telehealth care in a tertiary care IBD center. METHODS Pts seen at a tertiary care IBD center from January 2018 to March 2020 and August 2020 to January 2023 were included. Pts seen between March 2020 and July 2020 were excluded as the institution had mandatory telehealth visits during this period. Data was obtained from the Electronic Medical Record after IRB approval. Office visit types were based on billing codes, and office visits were only included in the analysis if seen by an MD or Advanced practice provider (APP). Drive time in minutes (min) was established from pt addresses using Esri geographic information systems (GIS) data, services and software. All new pts seen in the IBD Center are required to be seen in-clinic. After August 2020, pts seen in follow up can be seen at an in-clinic visit or via telehealth at the patient’s preference. Statistical analysis included Pearson Chi Square analysis for categorical variables and z-tests for means comparisons for continuous variables. RESULTS A total of 7,040 pts included. 56% were female. 89% were white, 7% Black or African American, and 76% had commercial health insurance, 17% had Medicare, 4% Medicaid, and 3% had no health insurance. New pts seen in the post-Covid period had shorter drive times than those seen in the pre-Covid period. (Table 1) Pts who had at least one year of follow up were more likely to be seen as a return visit in the post-Covid period vs the pre-Covid period (84% vs 76%, p<0.05). In the post-Covid period, 62% (3,575/5,750) pts had return IBD care by their MD or APP team delivered via telehealth. Pts seen in telehealth had significantly longer drive times than in-clinic-only pts (median 56.3 min [IQR 93.3] vs median 47.9 min [IQR 89.7]), p<0.001. (Table 2) This was significant for both the telehealth pts who had been seen in-clinic in the pre-Covid period as well as pts who were seen only in the post-Covid period. Pts who continued care only with in-clinic visits in the post-Covid period had significantly shorter drive times than pts who were seen in-clinic pre-Covid (38.8 min [IQR 60] vs 54.2 min [IQR 89.9]), p<0.001. CONCLUSION Telehealth is an effective strategy for continued multidisciplinary care at a tertiary care IBD center. More than half of patients continuing to participate in telehealth for follow up care, and patients were more likely to follow up in the post-Covid period. Patients who live farther away are more likely to continue IBD care via telehealth. Table 1 Overall Drive Time for New patients in the Pre- and Post-COVID periods Table 2 Drive Times for Return patients in the Pre-Covid and Post-Covid periods

We appreciate the thoughtful letter by Dr. Kannry regarding our paper, “Clinician Collaboration to Improve Clinical Decision Support: The Clickbusters Initiative.”¹ In his letter, Dr. Kannry highlights the distinction between medication decision support (MDS) and clinical decision support (CDS) and asserts that analyses of overrides between the 2 may not be comparable. We acknowledge the difference between the 2 types of CDS, but we respectfully disagree with the size of the gap in override rates. Epic provides median and quartile rates for its organizations across more than 800 metrics for benchmarking, including medication warnings (ie, MDS) and BestPractice Advisories (BPAs, ie, CDS). During May 2023, in the inpatient setting, interruptive medication warnings and BPAs had a median override or nonacceptance rate of 87.05% and 89.05%, respectively, and in the outpatient settings, the rates were 88.64% and 87.56%.² We wholeheartedly agree with Dr. Kannry’s concern about the lack of standardization for CDS measurement and benchmarking. We have seen, in our own work, how differences in the way that CDS measures are operationalized can lead to large differences in even simple measures like alert firing and acceptance rate. In 1 analysis, we reviewed MDS alerts during a 1-month period across 2 institutions and found that alert firing rates differed by more than 60% when comparing unique alerts and total alerts. Similarly, override rates also differed when considering total override responses (66.5%, 78.7%), initial overrides (62.3%, 77.9%), and overrides where medication orders were not discontinued within 24 h (50.7%, 62.8%).

Background: Clinical trials indicate continuous glucose monitor (CGM) use may benefit adults with type 2 diabetes, but CGM rates and correlates in real-world care settings are unknown. Objective: We sought to ascertain prevalence and correlates of CGM use and to examine rates of new CGM prescriptions across clinic types and medication regimens. Design: Retrospective cohort using electronic health records in a large academic medical center in the Southeastern US. Participants: Adults with type 2 diabetes and a primary care or endocrinology visit during 2021. Main measures: Age, gender, race, ethnicity, insurance, clinic type, insulin regimen, hemoglobin A1c values, CGM prescriptions, and prescribing clinic type. Key results: Among 30,585 adults with type 2 diabetes, 13% had used a CGM. CGM users were younger and more had private health insurance (p < .05) as compared to non-users; 72% of CGM users had an intensive insulin regimen, but 12% were not taking insulin. CGM users had higher hemoglobin A1c values (both most recent and most proximal to the first CGM prescription) than non-users. CGM users were more likely to receive endocrinology care than non-users, but 23% had only primary care visits in 2021. For each month in 2021, a mean of 90.5 (SD 12.5) people started using CGM. From 2020 to 2021, monthly rates of CGM prescriptions to new users grew 36% overall, but 125% in primary care. Most starting CGM in endocrinology had an intensive insulin regimen (82% vs. 49% starting in primary care), whereas 28% starting CGM in primary care were not using insulin (vs. 5% in endocrinology). Conclusion: CGM uptake for type 2 diabetes is increasing rapidly, with most growth in primary care. These trends present opportunities for healthcare system adaptations to support CGM use and related workflows in primary care to support growth in uptake.