Abstract:Objective: To propose a novel approach for enhancing clinical prediction models by combining structured and unstructured data with multimodal data fusion. Methods: We presented a comprehensive framework that integrated multimodal data sources, including textual clinical notes, structured electronic health records (EHRs), and relevant clinical data from National Electronic Injury Surveillance System (NEISS) datasets. We proposed a novel hybrid fusion method, which incorporated state-of-the-art pre-trained language model, to integrate unstructured clinical text with structured EHR data and other multimodal sources, thereby capturing a more comprehensive representation of patient information. Results: The experimental results demonstrated that the hybrid fusion approach significantly improved the performance of clinical prediction models compared to traditional fusion frameworks and unimodal models that rely solely on structured data or text information alone. The proposed hybrid fusion system with RoBERTa language encoder achieved the best prediction of the Top 1 injury with an accuracy of 75.00% and Top 3 injuries with an accuracy of 93.54%. Conclusion: Our study highlights the potential of integrating natural language processing (NLP) techniques with multimodal data fusion for enhancing clinical prediction models' performances. By leveraging the rich information present in clinical text and combining it with structured EHR data, the proposed approach can improve the accuracy and robustness of predictive models. The approach has the potential to advance clinical decision support systems, enable personalized medicine, and facilitate evidence-based health care practices. Future research can further explore the application of this hybrid fusion approach in real-world clinical settings and investigate its impact on improving patient outcomes.

Feature importance to explain multimodal prediction models. A clinical use case

Explainable Machine-Learning Predictions for Complications after Pediatric Congenital Heart Surgery

Understanding Risk Factors for Postoperative Mortality in Neonates Based on Explainable Machine Learning Technology

Multimodal risk prediction with physiological signals, medical images and clinical notes

Shedding Light on the Black Box: Explaining Deep Neural Network Prediction of Clinical Outcomes

Dynamic Predictions of Postoperative Complications from Explainable, Uncertainty-Aware, and Multi-Task Deep Neural Networks

Interpretable Neural Networks for Predicting Mortality Risk using Multi-modal Electronic Health Records

Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models

Explainable Prediction of Adverse Outcomes Using Clinical Notes

Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?

Interpretable Outcome Prediction with Sparse Bayesian Neural Networks in Intensive Care

Multimodal fusion models for pulmonary embolism mortality prediction

Self-explaining Hierarchical Model for Intraoperative Time Series

Generating Post-Hoc Explanation from Deep Neural Networks for Multi-Modal Medical Image Analysis Tasks

Optimizing Cardiac Surgery Risk Prediction: an Machine Learning Approach with Counterfactual Explanations.

Risk Prediction for Non-cardiac Surgery Using the 12-Lead Electrocardiogram: An Explainable Deep Learning Approach

Interpretable (not just posthoc-explainable) medical claims modeling for discharge placement to reduce preventable all-cause readmissions or death

Mixed-variable graphical modeling framework towards risk prediction of hospital-acquired pressure injury in spinal cord injury individuals

Comparative analysis of explainable machine learning prediction models for hospital mortality

XAI for In-hospital Mortality Prediction via Multimodal ICU Data