Abstract:Abstract Background and Aims Despite the rise in publications reporting Artificial Intelligence (AI) solutions in healthcare, their usage in clinical practice remains limited. Prominent barriers for their implementation in clinical practice are the lack of integration of AI-based tools in the hospital health information system and clinical workflows. We previously developed an AVF Failure risk score to assess the expected incidence of AVF failures within 3 months based on routinely collected dialysis data. The risk score showed high accuracy (AUC = 0.81) and calibration. Based on this risk score, we developed an AI-enhanced dashboard supporting the management of AVF in hemodialysis patients. The usability and usefulness of the AI-enhanced AVF management dashboard was evaluated in a pilot quality improvement program. Method A quality improvement framework was used to evaluate the usability of an AI-enhanced dashboard for AVF management (Fig. 1A). First, healthcare professionals (HCP) were involved to provide inputs for model design, endpoint definition, and potential predictors through various qualitative interview techniques such as brainstorming sessions and focus groups (Phase I). A machine learning model was then developed to assess the expected incidence of AVF failure within 3 months. The model used routinely collected clinical data to minimize additional burden on clinical workflow. (Phase II). After developing and validating the model, a model error analysis was conducted using case studies methodology to improve model performance (Phase III). Once the model reached a good accuracy, think-aloud interview was used to abstract HCP needs and to design a user interface for model output. A user-friendly dashboard was developed to display the risk of AVF failure within 3-months and patient's vascular access history (Phase IV). This solution was integrated in the health record system and updated monthly with the latest risk score and medical information. A pilot quality improvement program was conducted in 15 clinics across 4 countries (IT, CZ, SK, SG) from May to December 2022. HCP were asked to evaluate their agreement with the risk score using a 5-point Likert scale (completely disagree to agree) and assess the usefulness of the dashboard in clinical decision making. At this stage the dashboard is not used for medical decision making (Phase V). The evaluation results were compiled into a monthly report and shared with all the HCP involved in the project to promote awareness and engagement. After 3 months of usage, a round-table discussion was held with all participants to gather their feedback and discuss clinical cases (Phase VI). Results Access to the AI-enhanced dashboard increased during the evaluation period. In the first two months of the pilot quality improvement program, the AI-enhanced dashboard was evaluated for 20% patients with an AVF, a proportion that doubled by the end of the program (Fig. 1B). HCP reported 90% agreement with risk model rating (Fig. 1C), suggesting that the model capture relevant physiological status of the patient and its output is understandable by the healthcare staff. Qualitative analysis of round-table discussions showed that the dashboard is perceived as useful and clear. HCP suggested few changes in the user interface to enhance actionability and improve clinical workflow integration. Conclusion Our results support the usefulness and intelligibility of the AI-enhanced AVF management dashboard. The advance of AI-models highlighted the importance to develop new strategies for their integration and evaluation in clinical practice. The engagement of HCP in the early phases of AI-model design and development grants a continuous feedback loop bringing clinical and AI world together. A clinical perspective in dashboard design reflects the needs of healthcare professionals and enhance consideration of patient medical needs. The synergy between theoretical and clinical perspective is crucial to improve model performance, usability, and implementation in clinical practice.

Improving Patient Safety: Integrating Data Visualization and Communication Into Icu Workflow to Reduce Cognitive Load

Design and Practice of the Clinical Information System for ICU

The Impact of Visualization Dashboards on Quality of Care and Clinician Satisfaction: Integrative Literature Review

A Novel Continuous Real-Time Vital Signs Viewer for Intensive Care Units: Design and Evaluation Study

Centralized Multi-Patient Dashboards’ Impact on ICU Clinician Performance and Satisfaction: A Systematic Review

AI-Enhanced Intensive Care Unit: Revolutionizing Patient Care with Pervasive Sensing

User Perceptions of Avatar-Based Patient Monitoring for Intensive Care Units: An International Exploratory Sequential Mixed-Methods Study

Evaluation of an integrated intensive care unit monitoring display by critical care fellow physicians

Data Representation Structure to Support Clinical Decision-Making in the Pediatric Intensive Care Unit: Interview Study and Preliminary Decision Support Interface Design

Leveraging Computer Vision in the Intensive Care Unit (ICU) for Examining Visitation and Mobility

#3226 DEVELOPMENT AND USABILITY ASSESSMENT OF AN AI-ENHANCED DASHBOARD SUPPORTING AVF MANAGEMENT IN CLINICAL PRACTICE

A Perioperative Care Display for Understanding High Acuity Patients

Improving the Efficiency and Ease of Healthcare Analysis Through Use of Data Visualization Dashboards

Situation Awareness-Oriented Patient Monitoring with Visual Patient Technology: A Qualitative Review of the Primary Research

Visual TASK: A Collaborative Cognitive Aid for Acute Care Resuscitation

Anatomically Integrated In-Place Visualization of Patient Data for Cooperative Tasks with a Case Study on a Neurosurgical Ward

A novel tool for patient data management in the ICU-Ensuring timely and accurate vital data exchange among ICU team members

Improving preparation in the emergency trauma room: the development and impact of real-time data transfer and dashboard visualization system

ClinicalPath: a Visualization tool to Improve the Evaluation of Electronic Health Records in Clinical Decision-Making

Bringing Modern Machine Learning into Clinical Practice Through the Use of Intuitive Visualization and Human–Computer Interaction

Development of an Interactive Dashboard to Analyze Cognitive Workload of Surgical Teams During Complex Procedural Care