Abstract:Objective: To propose a novel approach for enhancing clinical prediction models by combining structured and unstructured data with multimodal data fusion. Methods: We presented a comprehensive framework that integrated multimodal data sources, including textual clinical notes, structured electronic health records (EHRs), and relevant clinical data from National Electronic Injury Surveillance System (NEISS) datasets. We proposed a novel hybrid fusion method, which incorporated state-of-the-art pre-trained language model, to integrate unstructured clinical text with structured EHR data and other multimodal sources, thereby capturing a more comprehensive representation of patient information. Results: The experimental results demonstrated that the hybrid fusion approach significantly improved the performance of clinical prediction models compared to traditional fusion frameworks and unimodal models that rely solely on structured data or text information alone. The proposed hybrid fusion system with RoBERTa language encoder achieved the best prediction of the Top 1 injury with an accuracy of 75.00% and Top 3 injuries with an accuracy of 93.54%. Conclusion: Our study highlights the potential of integrating natural language processing (NLP) techniques with multimodal data fusion for enhancing clinical prediction models' performances. By leveraging the rich information present in clinical text and combining it with structured EHR data, the proposed approach can improve the accuracy and robustness of predictive models. The approach has the potential to advance clinical decision support systems, enable personalized medicine, and facilitate evidence-based health care practices. Future research can further explore the application of this hybrid fusion approach in real-world clinical settings and investigate its impact on improving patient outcomes.

Multimodal Data Matters: Language Model Pre-Training Over Structured and Unstructured Electronic Health Records

Two heads are better than one: Enhancing medical representations by pre-training over structured and unstructured electronic health records

Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives

Multimodal risk prediction with physiological signals, medical images and clinical notes

Contrastive Learning on Multimodal Analysis of Electronic Health Records

Multi-Modal Perceiver Language Model for Outcome Prediction in Emergency Department

Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook

How to Leverage Multimodal EHR Data for Better Medical Predictions?

UniMed: Multimodal Multitask Learning for Medical Predictions.

That's the Wrong Lung! Evaluating and Improving the Interpretability of Unsupervised Multimodal Encoders for Medical Data

MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large Language Models

Global Contrastive Training for Multimodal Electronic Health Records with Language Supervision

M$^3$Care: Learning with Missing Modalities in Multimodal Healthcare Data

Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigm

Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM

From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice

Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks

REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models

Graph and text multi-modal representation learning with momentum distillation on Electronic Health Records

Research on Multimodal Fusion of Temporal Electronic Medical Records