Abstract:With the emergence of multimodal electronic health records, the evidence for diseases, events, or findings may be present across multiple modalities ranging from clinical to imaging and genomic data. Developing effective patient-tailored therapeutic guidance and outcome prediction will require fusing evidence across these modalities. Developing general-purpose frameworks capable of modeling fine-grained and multi-faceted complex interactions, both within and across modalities is an important open problem in multimodal fusion. Generalized multimodal fusion is extremely challenging as evidence for outcomes may not be uniform across all modalities, not all modality features may be relevant, or not all modalities may be present for all patients, due to which simple methods of early, late, or intermediate fusion may be inadequate. In this paper, we present a novel approach that uses the machinery of multiplexed graphs for fusion. This allows for modalities to be represented through their targeted encodings. We model their relationship between explicitly via multiplexed graphs derived from salient features in a combined latent space. We then derive a new graph neural network for multiplex graphs for task-informed reasoning. We compare our framework against several state-of-the-art approaches for multi-graph reasoning and multimodal fusion. As a sanity check on the neural network design, we evaluate the multiplexed GNN on two popular benchmark datasets, namely the AIFB and the MUTAG dataset against several state-of-the-art multi-relational GNNs for reasoning. Second, we evaluate our multiplexed framework against several state-of-the-art multimodal fusion frameworks on two large clinical datasets for two separate applications. The first is the NIH-TB portals dataset for treatment outcome prediction in Tuberculosis, and the second is the ABIDE dataset for Autism Spectrum Disorder classification. Through rigorous experimental evaluation, we demonstrate that the multiplexed GNN provides robust performance improvements in all of these diverse applications.

Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM

Multimodal risk prediction with physiological signals, medical images and clinical notes

Fusion of medical imaging and electronic health records with attention and multi-head machanisms

Combining structured and unstructured data for predictive models: a deep learning approach

REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models

MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large Language Models

Research on Multimodal Fusion of Temporal Electronic Medical Records

EMERGE: Integrating RAG for Improved Multimodal EHR Predictive Modeling

DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency

Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models

How to Leverage Multimodal EHR Data for Better Medical Predictions?

Fusing Modalities by Multiplexed Graph Neural Networks for Outcome Prediction from Medical Data and Beyond

Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives

Automated Fusion of Multimodal Electronic Health Records for Better Medical Predictions

Multimodal fusion network for ICU patient outcome prediction

Multi-task Heterogeneous Graph Learning on Electronic Health Records

HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling

Multimodal Data Matters: Language Model Pre-Training Over Structured and Unstructured Electronic Health Records

EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generation

Multimodal Clinical Pseudo-notes for Emergency Department Prediction Tasks using Multiple Embedding Model for EHR (MEME)

Multi-relational EHR representation learning with infusing information of Diagnosis and Medication