Aaron Chase’s research while affiliated with Emory University and other places

Background: Large language models (LLMs) have shown capability in diagnosing complex medical cases and passing medical licensing exams, but to date, only limited evaluations have studied how LLMs interpret, analyze, and optimize complex medication regimens. The purpose of this evaluation was to test four LLMs ability to identify medication errors and appropriate medication interventions on complex patient cases from the intensive care unit (ICU). Methods: A series of eight patient cases were developed by critical care pharmacists including history of present illness, laboratory values, vital signs, and medication regimens. Then, four LLMs (ChatGPT (GPT-3.5), ChatGPT (GPT-4), Claude2, and Llama2-7b) were prompted to develop a medication regimen for the patient. LLM generated medication regimens were then reviewed by a panel of seven critical care pharmacists to assess for presence of medication errors and clinical relevance. For each medication regimen recommended by the LLM, clinicians were asked to assess for if they would continue a medication, identify perceived medication errors in the medications recommended, identify the presence of life-threatening medication choices, and rank overall agreement on a 5-point Likert scale. Results: The clinician panel rated to continue therapies recommended by the LLMs between 55.8-67.9% of the time. Clinicians perceived between 1.57-4.29 medication errors per recommended regimen, and life-threatening recommendations were present between 15.0-55.3% of the time. Level agreement was between 1.85-2.67 for the four LLMs. Conclusions: LLMs demonstrated potential to serve as clinical decision support for the management of complex medication regimens with further domain specific training; however, caution should be used when employing LLMs for medication management given the present capabilities.

Background: Large language models (LLMs) such as ChatGPT have emerged as promising artificial intelligence tools to support clinical decision making. The ability of ChatGPT to evaluate medication regimens, identify drug-drug interactions (DDIs), and provide clinical recommendations is unknown. The purpose of this study is to examine the performance of GPT-4 to identify clinically relevant DDIs and assess accuracy of recommendations provided. Methods: A total of 15 medication regimens were created containing commonly encountered DDIs that were considered either clinically significant or clinically unimportant. Two separate prompts were developed for medication regimen evaluation. The primary outcome was if GPT-4 identified the most relevant DDI within the medication regimen. Secondary outcomes included rating GPT-4s interaction rationale, clinical relevance ranking, and overall clinical recommendations. Interrater reliability was determined using kappa statistic. Results: GPT-4 identified the intended DDI in 90% of medication regimens provided (27/30). GPT-4 categorized 86% as highly clinically relevant compared to 53% being categorized as highly clinically relevant by expert opinion. Inappropriate clinical recommendations potentially causing patient harm were provided in 14% of responses provided by GPT-4 (2/14), and 63% of responses contained accurate information but incomplete recommendations (19/30). Conclusions: While GPT-4 demonstrated promise in its ability to identify clinically relevant DDIs, application to clinical cases remains an area of investigation. Findings from this study may assist in future development and refinement of LLMs for drug-drug interaction queries to assist in clinical decision-making.

Disclaimer In an effort to expedite the publication of articles, AJHP is posting manuscripts online as soon as possible after acceptance. Accepted manuscripts have been peer-reviewed and copyedited, but are posted online before technical formatting and author proofing. These manuscripts are not the final version of record and will be replaced with the final article (formatted per AJHP style and proofed by the authors) at a later time. Purpose To familiarize clinicians with the emerging concepts in critical care research of Bayesian thinking and personalized medicine through phenotyping and explain their clinical relevance by highlighting how they address the issues of frequent negative trials and heterogeneity of treatment effect. Summary The past decades have seen many negative (effect-neutral) critical care trials of promising interventions, culminating in calls to improve the field’s research through adopting Bayesian thinking and increasing personalization of critical care medicine through phenotyping. Bayesian analyses add interpretive power for clinicians as they summarize treatment effects based on probabilities of benefit or harm, contrasting with conventional frequentist statistics that either affirm or reject a null hypothesis. Critical care trials are beginning to include prospective Bayesian analyses, and many trials have undergone reanalysis with Bayesian methods. Phenotyping seeks to identify treatable traits to target interventions to patients expected to derive benefit. Phenotyping and subphenotyping have gained prominence in the most syndromic and heterogenous critical care disease states, acute respiratory distress syndrome and sepsis. Grouping of patients has been informative across a spectrum of clinically observable physiological parameters, biomarkers, and genomic data. Bayesian thinking and phenotyping are emerging as elements of adaptive clinical trials and predictive enrichment, paving the way for a new era of high-quality evidence. These concepts share a common goal, sifting through the noise of heterogeneity in critical care to increase the value of existing and future research. Conclusion The future of critical care medicine will inevitably involve modification of statistical methods through Bayesian analyses and targeted therapeutics via phenotyping. Clinicians must be familiar with these systems that support recommendations to improve decision-making in the gray areas of critical care practice.

OBJECTIVES/GOALS: To determine if incorporating specific laboratory values and plasma biomarkers (club cell secretory protein (CC16), matrix metalloproteinase 3 (MMP3), interleukin 8 (IL-8), protein C) to the Lung Injury Prediction (LIP) Score improves the predictive value for development of acute respiratory distress syndrome (ARDS) in ICU patients. METHODS/STUDY POPULATION: Adult patients admitted to the ICU on supplemental oxygen over baseline requirement with a LIP Score ≥6 will be included. Patients admitted to the ICU >24 hours, end-stage renal disease, decompensated heart failure, or <100 µL plasma available will be excluded. Whole blood will be collected from the core lab, centrifuged, and plasma will be stored at -80°C. Protein biomarkers will be measured using enzyme-linked immunosorbent assay. Baseline characteristics, laboratory values, ventilator parameters, and clinical outcomes will be collected from the medical record. ARDS will be defined by the Berlin criteria. Machine learning methods will be used to identify the model with the highest predictive accuracy. Area under the receiver operating characteristic curve of each model will be compared to the LIP Score. RESULTS/ANTICIPATED RESULTS: Research is in progress. Plasma samples and clinical data have been collected for 148 of the 160 samples required to achieve power. Biomarker analysis will take place after sample collection is complete. We anticipate a machine learning model incorporating laboratory values and one or more plasma biomarkers into the LIP Score will outperform the baseline LIP Score for prediction of ARDS development. DISCUSSION/SIGNIFICANCE: Delayed diagnosis and intervention contribute to poor ARDS outcomes. Current predictive models for ARDS have low accuracy and enriching these models with plasma biomarkers may increase their predictive value. Development of accurate models may facilitate earlier ARDS diagnosis and intervention as well as enrichment strategies for ARDS trials.

Purpose: Critical care pharmacists are considered essential members of the healthcare team; however, justification and recruitment of new positions, especially in the evening or weekend shifts, remains a significant challenge. The purpose of this study was to investigate the number of interventions, type of interventions, and associated cost savings with the addition of 1 board certified critical care clinical pharmacist to evening shift. Methods: This was a prospective collection and characterization of 1 evening shift critical care pharmacist’s clinical interventions over a 12-week period. Interventions were collected and categorized daily from 13:00 to 22:00 Monday through Friday. After collection was complete, cost savings estimates were calculated using pharmacy wholesaler acquisition cost. Results: Interventions were collected on 52 of 60 weekdays. A total of 510 interventions were collected with an average of 9.8 interventions accepted per day. The most common interventions included transitions of care, medication dose adjustment, and antibiotic de-escalation and the highest proportion of interventions occurred in the medical intensive care unit. An estimated associated cost avoidance of $66 537.80 was calculated for an average of$1279.57 saved per day. Additionally, 22 (4.1%) of interventions were considered high yield interventions upon independent review by 2 pharmacists. Conclusion: The addition of 1 board-certified critical care pharmacist to evening shift resulted in multiple interventions across several categories and a significant cost avoidance when calculated using conservative measures.

Purpose: The purpose of this study was to determine the relationship between medication regimen complexity-intensive care unit (MRC-ICU) score at 24 hours and medication errors identified throughout the ICU. Methods: A single-center, observational study was conducted from August to October 2021. The primary outcome was the association between MRC-ICU at 24 hours and total medication errors identified. During the prospective component, ICU pharmacists recorded medication errors identified over an 8-week period. During the retrospective component, the electronic medical record was reviewed to collect patient demographics, outcomes, and MRC-ICU score at 24 hours. The primary outcome of the relationship of MRC-ICU at 24 hours to medication errors was assessed using Pearson correlation. Results: A total of 150 patients were included. There were 2 pharmacists who recorded 634 errors during the 8-week study period. No significant relationship between MRC-ICU and medication errors was observed (r² = .13, P = .11). Exploratory analyses of MRC-ICU relationship to major interventions and harm scores showed that MRC-ICU scores >10 had more major interventions (27 vs 14, P = .27) and higher harm scores (15 vs 7, P = .33), although these values were not statistically significant. Conclusion: Medication errors appear to occur independently of medication regimen complexity. Critical care pharmacists were responsible for mitigating a large number of medication errors.

Purpose The medication regimen complexity-intensive care unit (MRC-ICU) score was developed prior to the existence of COVID-19. The purpose of this study was to assess if MRC-ICU could predict in-hospital mortality in patients with COVID-19. Methods A single-center, observational study was conducted from August 2020 to January 2021. The primary outcome of this study was the area under the receiver operating characteristic (AUROC) for in-hospital mortality for the 48-hour MRC-ICU. Age, sequential organ failure assessment (SOFA), and World Health Organization (WHO) COVID-19 Severity Classification were assessed. Logistic regression was performed to predict in-hospital mortality as well as WHO Severity Classification at 7 days. Results A total of 149 patients were included. The median SOFA score was 8 (IQR 5-11) and median MRC-ICU score at 48 hours was 15 (IQR 7-21). The in-hospital mortality rate was 36% (n = 54). The AUROC for MRC-ICU was 0.71 (95% Confidence Interval (CI), 0.62-0.78) compared to 0.66 for age, 0.81 SOFA, and 0.72 for the WHO Severity Classification. In univariate analysis, age, SOFA, MRC-ICU, and WHO Severity Classification all demonstrated significant association with in-hospital mortality, while SOFA, MRC-ICU, and WHO Severity Classification demonstrated significant association with WHO Severity Classification at 7 days. In univariate analysis, all 4 characteristics showed significant association with mortality; however, only age and SOFA remained significant following multivariate analysis. Conclusion In the first analysis of medication-related variables as a predictor of severity and in-hospital mortality in COVID-19, MRC-ICU demonstrated acceptable predictive ability as represented by AUROC; however, SOFA was the strongest predictor in both AUROC and regression analysis.

Background Matrix metalloproteinase-3 (MMP-3) is a proteolytic enzyme involved in acute respiratory distress syndrome (ARDS) pathophysiology that may serve as a lung-specific biomarker in ARDS. Methods This study was a secondary biomarker analysis of a subset of Albuterol for the Treatment of Acute Lung Injury (ALTA) trial patients to determine the prognostic value of MMP-3. Plasma sample MMP-3 was measured by enzyme-linked immunosorbent assay. The primary outcome was the area under the receiver operating characteristic (AUROC) of MMP-3 at day 3 for the prediction of 90-day mortality. Results A total of 100 unique patient samples were evaluated and the AUROC analysis of day three MMP-3 showed an AUROC of 0.77 for the prediction of 90-day mortality (95% confidence interval: 0.67–0.87), corresponding to a sensitivity of 92% and specificity of 63% and an optimal cutoff value of 18.4 ng/mL. Patients in the high MMP-3 group (≥ 18.4 ng/mL) showed higher mortality compared to the non-elevated MMP-3 group (< 18.4 ng/mL) (47% vs. 4%, p < 0.001). A positive difference in day zero and day three MMP-3 concentration was predictive of mortality with an AUROC of 0.74 correlating to 73% sensitivity, 81% specificity, and an optimal cutoff value of + 9.5 ng/mL. Conclusions Day three MMP-3 concentration and difference in day zero and three MMP-3 concentrations demonstrated acceptable AUROCs for predicting 90-day mortality with a cut-point of 18.4 ng/mL and + 9.5 ng/mL, respectively. These results suggest a prognostic role of MMP-3 in ARDS.