Abstract:Outcomes 1. Understand how an AI model can augment operational, workflow, and technology redesign to support improvements in serious illness discussions, documentation, and ACP billing. 2. Apply evaluation tools to determine whether an AI model might be appropriate for their clinical context and discuss how an AI mortality model can augment clinician intelligence to improve goal concordant care delivery. Key Message A NCI-designated Cancer Center incorporated an AI model to significantly improve goals of care (GOC) discussions, documentation, and ACP billing with seriously ill patients. The AI model was helpful in identifying almost 60% of priority patients in 2022 and was important in the overall strategy to improve goal concordant care. Importance Advances in therapy within cancer including targeted immunotherapy and CAR-T cell therapies have made it increasingly difficult to predict prognosis. AI risk stratification models may help prioritize the limited capacity of overstretched clinicians to identify patients who would maximally benefit from goals of care discussions. Objectives To provide background on an AI model that outperformed medical oncologists in predicting 90-day mortality for patients with metastatic solid tumors1, provide recommendations on considerations prior to implementing AI within an institution, and show how an NCI-designated Cancer Center incorporated AI within clinical workflows to increase goals of care discussions. Scientific Methods Utilized With broad institutional endorsement, focused multi-disciplinary groups redesigned workflows and the EHR to support automated identification and engagement of advanced cancer patients. Significant EHR customizations were made to automate priority patient identification, including integration of a supervised machine learning 90-day mortality model and evaluation against department-specific clinical criteria. Visual indicators, nudges, and an easy-to-use GOC template with embedded ACP billing were also introduced to improve visibility of priority patients' GOC statuses and ease administrative burden. Results In 2022, 867 priority patients were identified (59% by AI model). Amongst all services, use of ACP billing codes (99497 & 99498) increased 10-fold between 2020 as a baseline year and 2022 (268 to 2650). GOC documentation increased over 7-fold (738 to 5660). Workflows, EHR screenshots, and Tableau dashboards will be shared to demonstrate how AI was incorporated to augment clinicians within workflow. Conclusion(s) AI can be beneficial in supporting a multi-disciplinary effort to implement an Improving Goal Concordant Care Initiative. This led to significant increases in GOC discussions captured within structured documentation in the EHR and through use of ACP billing codes. Impact AI risk stratification can assist goals of care workflows and improve the volume of goals of care discussions to align patient preferences with care. Further work is needed to understand how clinician augmented intelligence can improve performance of risk stratification systems. Keywords Communication and prognostication/Palliative care in oncology

Abstract 350: Measuring the Return on Investment of an Artificial Intelligence (AI) Early Warning System

Abstract 15139: Applying an Artificial Intelligence-Enabled Electrocardiographic System for Reducing Mortality: A Pragmatic Randomized Clinical Trial

The Impact of a Machine Learning Early Warning Score on Hospital Mortality: A Multicenter Clinical Intervention Trial

AI-enabled electrocardiography alert intervention and all-cause mortality: a pragmatic randomized clinical trial

Evaluation of the Impact of a 90-day Mortality Prediction Model Integrated in the EHR for Advanced Care Planning

Effectiveness of an Artificial Intelligence–Enabled Intervention for Detecting Clinical Deterioration

Abstract 15437: Use of Artificial Intelligence to Decrease Data Deluge From Remote Monitoring of Cardiac Implantable Electronic Devices

The Kaiser Permanente Northern California Advance Alert Monitor Program: An Automated Early Warning System for Adults at Risk for In-Hospital Clinical Deterioration

Using Electronic Health Record Collected Clinical Variables to Predict Medical Intensive Care Unit Mortality

Multicenter Development and Prospective Validation of eCARTv5: A Gradient Boosted Machine Learning Early Warning Score

Using AI to Improve Goal Concordant Care: Deep Dive into a 90-Day Mortality Model and Incorporation into Clinical Care (CO202C)

Estimation of the maximum potential cost saving from reducing serious adverse events in hospitalized patients

Artificial intelligence-enabled electrocardiogram for mortality and cardiovascular risk estimation: a model development and validation study

Artificial Intelligence Driven Prehospital ECG Interpretation for the Reduction of False Positive Emergent Cardiac Catheterization Lab Activations: A Retrospective Cohort Study

Less is more: Detecting clinical deterioration in the hospital with machine learning using only age, heart rate, and respiratory rate

Longitudinal evaluation of an electronic audit and feedback system for patient safety in a large tertiary hospital setting

Assessment of the Feasibility of automated, real-time clinical decision support in the emergency department using electronic health record data

Clinical evaluation of a machine learning-based early warning system for patient deterioration

Impact of a Clinical Pharmacist-Led, Artificial Intelligence-Supported Medication Adherence Program on Medication Adherence Performance, Chronic Disease Control Measures, and Cost Savings

Improving ICU Risk Predictive Models Through Automation Designed for Resiliency Against Documentation Bias

Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study