Abstract:With the emergence of multimodal electronic health records, the evidence for diseases, events, or findings may be present across multiple modalities ranging from clinical to imaging and genomic data. Developing effective patient-tailored therapeutic guidance and outcome prediction will require fusing evidence across these modalities. Developing general-purpose frameworks capable of modeling fine-grained and multi-faceted complex interactions, both within and across modalities is an important open problem in multimodal fusion. Generalized multimodal fusion is extremely challenging as evidence for outcomes may not be uniform across all modalities, not all modality features may be relevant, or not all modalities may be present for all patients, due to which simple methods of early, late, or intermediate fusion may be inadequate. In this paper, we present a novel approach that uses the machinery of multiplexed graphs for fusion. This allows for modalities to be represented through their targeted encodings. We model their relationship between explicitly via multiplexed graphs derived from salient features in a combined latent space. We then derive a new graph neural network for multiplex graphs for task-informed reasoning. We compare our framework against several state-of-the-art approaches for multi-graph reasoning and multimodal fusion. As a sanity check on the neural network design, we evaluate the multiplexed GNN on two popular benchmark datasets, namely the AIFB and the MUTAG dataset against several state-of-the-art multi-relational GNNs for reasoning. Second, we evaluate our multiplexed framework against several state-of-the-art multimodal fusion frameworks on two large clinical datasets for two separate applications. The first is the NIH-TB portals dataset for treatment outcome prediction in Tuberculosis, and the second is the ABIDE dataset for Autism Spectrum Disorder classification. Through rigorous experimental evaluation, we demonstrate that the multiplexed GNN provides robust performance improvements in all of these diverse applications.

Gated multimodal networks

Gated Multimodal Units for Information Fusion

X-Gacmn: An X-Shaped Generative Adversarial Cross-Modal Network With Hypersphere Embedding

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

MMTM: Multimodal Transfer Module for CNN Fusion

Gated attention fusion network for multimodal sentiment classification

MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks

MSTGC: Multi-Channel Spatio-Temporal Graph Convolution Network for Multi-Modal Brain Networks Fusion

Fusing Modalities by Multiplexed Graph Neural Networks for Outcome Prediction from Medical Data and Beyond

Gated Value Network for Multilabel Classification.

Graph Metanetworks for Processing Diverse Neural Architectures

CentralNet: a Multilayer Approach for Multimodal Fusion

Hypercomplex Graph Neural Network: Towards Deep Intersection of Multi-modal Brain Networks

Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion

Pre-gating and contextual attention gate - A new fusion method for multi-modal data tasks

Generalized Multimodal Fusion via Poisson-Nernst-Planck Equation

Multimodal Sentiment Analysis Based on Cross-Modal Attention and Gated Cyclic Hierarchical Fusion Networks

Adaptive Fusion Techniques for Multimodal Data

Multiplex Graph Networks for Multimodal Brain Network Analysis

Multimodal Understanding Through Correlation Maximization and Minimization

Neural Dependency Coding inspired Multimodal Fusion