Abstract:Outcomes 1. Understand how an AI model can augment operational, workflow, and technology redesign to support improvements in serious illness discussions, documentation, and ACP billing. 2. Apply evaluation tools to determine whether an AI model might be appropriate for their clinical context and discuss how an AI mortality model can augment clinician intelligence to improve goal concordant care delivery. Key Message A NCI-designated Cancer Center incorporated an AI model to significantly improve goals of care (GOC) discussions, documentation, and ACP billing with seriously ill patients. The AI model was helpful in identifying almost 60% of priority patients in 2022 and was important in the overall strategy to improve goal concordant care. Importance Advances in therapy within cancer including targeted immunotherapy and CAR-T cell therapies have made it increasingly difficult to predict prognosis. AI risk stratification models may help prioritize the limited capacity of overstretched clinicians to identify patients who would maximally benefit from goals of care discussions. Objectives To provide background on an AI model that outperformed medical oncologists in predicting 90-day mortality for patients with metastatic solid tumors1, provide recommendations on considerations prior to implementing AI within an institution, and show how an NCI-designated Cancer Center incorporated AI within clinical workflows to increase goals of care discussions. Scientific Methods Utilized With broad institutional endorsement, focused multi-disciplinary groups redesigned workflows and the EHR to support automated identification and engagement of advanced cancer patients. Significant EHR customizations were made to automate priority patient identification, including integration of a supervised machine learning 90-day mortality model and evaluation against department-specific clinical criteria. Visual indicators, nudges, and an easy-to-use GOC template with embedded ACP billing were also introduced to improve visibility of priority patients' GOC statuses and ease administrative burden. Results In 2022, 867 priority patients were identified (59% by AI model). Amongst all services, use of ACP billing codes (99497 & 99498) increased 10-fold between 2020 as a baseline year and 2022 (268 to 2650). GOC documentation increased over 7-fold (738 to 5660). Workflows, EHR screenshots, and Tableau dashboards will be shared to demonstrate how AI was incorporated to augment clinicians within workflow. Conclusion(s) AI can be beneficial in supporting a multi-disciplinary effort to implement an Improving Goal Concordant Care Initiative. This led to significant increases in GOC discussions captured within structured documentation in the EHR and through use of ACP billing codes. Impact AI risk stratification can assist goals of care workflows and improve the volume of goals of care discussions to align patient preferences with care. Further work is needed to understand how clinician augmented intelligence can improve performance of risk stratification systems. Keywords Communication and prognostication/Palliative care in oncology

Evaluation of the Impact of a 90-day Mortality Prediction Model Integrated in the EHR for Advanced Care Planning

Using AI to Improve Goal Concordant Care: Deep Dive into a 90-Day Mortality Model and Incorporation into Clinical Care (CO202C)

Machine learning for targeted advance care planning in cancer patients: a quality improvement study

Abstract 350: Measuring the Return on Investment of an Artificial Intelligence (AI) Early Warning System

The Kaiser Permanente Northern California Advance Alert Monitor Program: An Automated Early Warning System for Adults at Risk for In-Hospital Clinical Deterioration

Improving ICU Risk Predictive Models Through Automation Designed for Resiliency Against Documentation Bias

‘ALERT’, a machine learning predictive model to identify six-month mortality in patients with advanced cancer: Perception of the utility and acceptability among patients.

Equity in Using Artificial Intelligence Mortality Predictions to Target Goals of Care Documentation

Improving Palliative Care with Deep Learning

Looking Beyond Mortality Prediction: Primary Care Physician Views of Patients’ Palliative Care Needs Predicted by a Machine Learning Tool

Implications of the Use of Artificial Intelligence Predictive Models in Health Care Settings : A Simulation Study

In-hospital mortality, readmission, and prolonged length of stay risk prediction leveraging historical electronic patient records

Identifying Facilitators and Barriers to Implementation of AI-Assisted Clinical Decision Support in an Electronic Health Record System

Leveraging patients' longitudinal data to improve the Hospital One-year Mortality Risk

Enhancing Prediction Models for One-Year Mortality in Patients with Acute Myocardial Infarction and Post Myocardial Infarction Syndrome

Closing the Gap in High-Risk Pregnancy Care Using Machine Learning and Human-AI Collaboration

Comparing Clinician Estimates versus a Statistical Tool for Predicting Risk of Death within 45 Days of Admission for Cancer Patients

Clinical evaluation of a machine learning-based early warning system for patient deterioration

Predicting Risk of Alzheimer’s Diseases and Related Dementias with AI Foundation Model on Electronic Health Records

Applying Artificial Intelligence to Address the Knowledge Gaps in Cancer Care

Predicting Risk of Alzheimer's Diseases and Related Dementias with AI Foundation Model on Electronic Health Records