Abstract:Objective: With the increasing amount and growing variety of healthcare data, multimodal machine learning supporting integrated modeling of structured and unstructured data is an increasingly important tool for clinical machine learning tasks. However, it is non-trivial to manage the differences in dimensionality, volume, and temporal characteristics of data modalities in the context of a shared target task. Furthermore, patients can have substantial variations in the availability of data, while existing multimodal modeling methods typically assume data completeness and lack a mechanism to handle missing modalities. Methods: We propose a Transformer-based fusion model with modality-specific tokens that summarize the corresponding modalities to achieve effective cross-modal interaction accommodating missing modalities in the clinical context. The model is further refined by inter-modal, inter-sample contrastive learning to improve the representations for better predictive performance. We denote the model as Attention-based cRoss-MOdal fUsion with contRast (ARMOUR). We evaluate ARMOUR using two input modalities (structured measurements and unstructured text), six clinical prediction tasks, and two evaluation regimes, either including or excluding samples with missing modalities. Results: Our model shows improved performances over unimodal or multimodal baselines in both evaluation regimes, including or excluding patients with missing modalities in the input. The contrastive learning improves the representation power and is shown to be essential for better results. The simple setup of modality-specific tokens enables ARMOUR to handle patients with missing modalities and allows comparison with existing unimodal benchmark results. Conclusion: We propose a multimodal model for robust clinical prediction to achieve improved performance while accommodating patients with missing modalities. This work could inspire future research to study the effective incorporation of multiple, more complex modalities of clinical data into a single model.

Efficient Multi-Model Fusion with Adversarial Complementary Representation Learning

Multi-level Multi-task Representation Learning with Adaptive Fusion for Multimodal Sentiment Analysis

Exploiting Multi-modal Fusion for Robust Face Representation Learning with Missing Modality

Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

Defending Multimodal Fusion Models against Single-Source Adversaries

Asymmetric Convolution: an Efficient and Generalized Method to Fuse Feature Maps in Multiple Vision Tasks

Fusing Models with Complementary Expertise

Contrastive Fusion Representation: Mitigating Adversarial Attacks on VQA Models

Dense Multimodal Fusion for Hierarchically Joint Representation

ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning

Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion

Acoustic Model Fusion for End-to-end Speech Recognition

A Self-Adjusting Fusion Representation Learning Model for Unaligned Text-Audio Sequences

AbFTNet: An Efficient Transformer Network with Alignment before Fusion for Multimodal Automatic Modulation Recognition

Dual-model self-regularization and fusion for domain adaptation of robust speaker verification

MCSFF: Multi-modal Consistency and Specificity Fusion Framework for Entity Alignment

Multi-Modal Fusion-Based Multi-Task Semantic Communication System

Attention-based multimodal fusion with contrast for robust clinical prediction in the face of missing modalities

Active Multiview Fusion Framework for SAR Automatic Target Recognition

Attention Bottlenecks for Multimodal Fusion

Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion