Abstract:With the emergence of multimodal electronic health records, the evidence for diseases, events, or findings may be present across multiple modalities ranging from clinical to imaging and genomic data. Developing effective patient-tailored therapeutic guidance and outcome prediction will require fusing evidence across these modalities. Developing general-purpose frameworks capable of modeling fine-grained and multi-faceted complex interactions, both within and across modalities is an important open problem in multimodal fusion. Generalized multimodal fusion is extremely challenging as evidence for outcomes may not be uniform across all modalities, not all modality features may be relevant, or not all modalities may be present for all patients, due to which simple methods of early, late, or intermediate fusion may be inadequate. In this paper, we present a novel approach that uses the machinery of multiplexed graphs for fusion. This allows for modalities to be represented through their targeted encodings. We model their relationship between explicitly via multiplexed graphs derived from salient features in a combined latent space. We then derive a new graph neural network for multiplex graphs for task-informed reasoning. We compare our framework against several state-of-the-art approaches for multi-graph reasoning and multimodal fusion. As a sanity check on the neural network design, we evaluate the multiplexed GNN on two popular benchmark datasets, namely the AIFB and the MUTAG dataset against several state-of-the-art multi-relational GNNs for reasoning. Second, we evaluate our multiplexed framework against several state-of-the-art multimodal fusion frameworks on two large clinical datasets for two separate applications. The first is the NIH-TB portals dataset for treatment outcome prediction in Tuberculosis, and the second is the ABIDE dataset for Autism Spectrum Disorder classification. Through rigorous experimental evaluation, we demonstrate that the multiplexed GNN provides robust performance improvements in all of these diverse applications.

Multimodal Prediction based on Graph Representations

Multimodal Vehicle Trajectory Prediction Based on Graph Convolutional Networks

Multimodal Graph for Unaligned Multimodal Sequence Analysis via Graph Convolution and Graph Pooling

Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion

Multi-modal Graph and Sequence Fusion Learning for Recommendation.

Multimodal Graph Learning for Cross-Modal Retrieval.

Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion

Multi-modal Recommendation Based on Knowledge Graph

Multimodal Topic Detection in Social Networks with Graph Fusion.

Efficient Low-rank Multimodal Fusion with Modality-Specific Factors

Fusing Modalities by Multiplexed Graph Neural Networks for Outcome Prediction from Medical Data and Beyond

Multimodal graph learning based on 3D Haar semi-tight framelet for student engagement prediction

IMF: Interactive Multimodal Fusion Model for Link Prediction

A Graph Convolution-Based Heterogeneous Fusion Network for Multimodal Sentiment Analysis

Multimodal learning based approaches for link prediction in social networks

Gated Multimodal Units for Information Fusion

Remote Sensing and Time Series Data Fused Multimodal Prediction Model Based on Interaction Analysis

Learning to Predict: A Fast Re-constructive Method to Generate Multimodal Embeddings

Multimodal Data Fusion In Text-Image Heterogeneous Graph For Social Media Recommendation

Attention-guided Multi-step Fusion: A Hierarchical Fusion Network for Multimodal Recommendation