Abstract:Objective: With the increasing amount and growing variety of healthcare data, multimodal machine learning supporting integrated modeling of structured and unstructured data is an increasingly important tool for clinical machine learning tasks. However, it is non-trivial to manage the differences in dimensionality, volume, and temporal characteristics of data modalities in the context of a shared target task. Furthermore, patients can have substantial variations in the availability of data, while existing multimodal modeling methods typically assume data completeness and lack a mechanism to handle missing modalities. Methods: We propose a Transformer-based fusion model with modality-specific tokens that summarize the corresponding modalities to achieve effective cross-modal interaction accommodating missing modalities in the clinical context. The model is further refined by inter-modal, inter-sample contrastive learning to improve the representations for better predictive performance. We denote the model as Attention-based cRoss-MOdal fUsion with contRast (ARMOUR). We evaluate ARMOUR using two input modalities (structured measurements and unstructured text), six clinical prediction tasks, and two evaluation regimes, either including or excluding samples with missing modalities. Results: Our model shows improved performances over unimodal or multimodal baselines in both evaluation regimes, including or excluding patients with missing modalities in the input. The contrastive learning improves the representation power and is shown to be essential for better results. The simple setup of modality-specific tokens enables ARMOUR to handle patients with missing modalities and allows comparison with existing unimodal benchmark results. Conclusion: We propose a multimodal model for robust clinical prediction to achieve improved performance while accommodating patients with missing modalities. This work could inspire future research to study the effective incorporation of multiple, more complex modalities of clinical data into a single model.

CMACF: Transformer-based Cross-Modal Attention Cross-Fusion Model for Systemic Lupus Erythematosus Diagnosis Combining Raman Spectroscopy, FTIR Spectroscopy, and Metabolomics

Diagnosis of systemic lupus erythematosus using cross-modal specific transfer fusion technology based on infrared spectra and metabolomics

Multimodal Fusion with Cross-attention Transformer for HCC Early Recurrence Prediction from Multi-Phase CT and Clinical Data

Transformer-Based Multi-Modal Data Fusion Method for COPD Classification and Physiological and Biochemical Indicators Identification

Cross Branch Co-Attention Network Multimodal Models Based on Raman and FTIR Spectroscopy for Diagnosis of Multiple Selected Cancers

MFCSA-CAT: a Multimodal Fusion Method for Cancer Survival Analysis Based on Cross-Attention Transformer

Transformer-based Cross-Modal Multi-Contrast Network for Ophthalmic Diseases Diagnosis

Missing-modality Enabled Multi-modal Fusion Architecture for Medical Data

SLE diagnosis research based on SERS combined with a multi-modal fusion method

Application of Multimodal Fusion Deep Learning Model in Disease Recognition

Interpretation on Deep Multimodal Fusion for Diagnostic Classification

Attention-based multimodal fusion with contrast for robust clinical prediction in the face of missing modalities

MACTFusion: Lightweight Cross Transformer for Adaptive Multimodal Medical Image Fusion

An Interpretable Cross-Attentive Multi-modal MRI Fusion Framework for Schizophrenia Diagnosis

Enhancing Heart Failure Diagnosis Through Multi-Modal Data Integration and Deep Learning

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis

Adversarial Multimodal Fusion with Attention Mechanism for Skin Lesion Classification Using Clinical and Dermoscopic Images

Research on Multimodal Fusion of Temporal Electronic Medical Records

Multimodal Fusion of Echocardiography and Electronic Health Records for the Detection of Cardiac Amyloidosis

Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI

TAMM: A Task-Adaptive Multi-Modal Fusion Network for Facial-Related Health Assessments on 3D Facial Images