Abstract:With the growing success of multi-modal learning, research on the robustness of multi-modal models, especially when facing situations with missing modalities, is receiving increased attention. Nevertheless, previous studies in this domain exhibit certain limitations, as they often lack theoretical insights or their methodologies are tied to specific network architectures or modalities. We model the scenarios of multi-modal models encountering missing modalities from an information-theoretic perspective and illustrate that the performance ceiling in such scenarios can be approached by efficiently utilizing the information inherent in non-missing modalities. In practice, there are two key aspects: (1) The encoder should be able to extract sufficiently good features from the non-missing modality; (2) The extracted features should be robust enough not to be influenced by noise during the fusion process across modalities. To this end, we introduce Uni-Modal Ensemble with Missing Modality Adaptation (UME-MMA). UME-MMA employs uni-modal pre-trained weights for the multi-modal model to enhance feature extraction and utilizes missing modality data augmentation techniques to better adapt to situations with missing modalities. Apart from that, UME-MMA, built on a late-fusion learning framework, allows for the plug-and-play use of various encoders, making it suitable for a wide range of modalities and enabling seamless integration of large-scale pre-trained encoders to further enhance performance. And we demonstrate UME-MMA's effectiveness in audio-visual datasets (e.g., AV-MNIST, Kinetics-Sound, AVE) and vision-language datasets (e.g., MM-IMDB, UPMC Food101).

Beyond unimodal regression: modelling multimodality with piecewise unimodal regression or deconvolution models

Similarity-Based Multimodal Regression

Multimodal Understanding Through Correlation Maximization and Minimization

Unimodal Cyclic Regularization for Training Multimodal Image Registration Networks.

Conformal Prediction for Multimodal Regression

Deep Multimodal Learning with Missing Modality: A Survey

Multinomial Restricted Unfolding

Nonparametric multimodal regression for circular data

Maximum Likelihood Estimation for Multimodal Learning with Missing Modality

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?

A Modular End-to-End Multimodal Learning Method for Structured and Unstructured Data

Orthogonalized Kernel Debiased Machine Learning for Multimodal Data Analysis

Integrative Factor Regression and Its Inference for Multimodal Data Analysis

Deformable Registration of Biomedical Images Using 2D Hidden Markov Models

Robust Multimodal Representation under Uncertain Missing Modalities

Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework

Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications

Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discovery

Understanding Unimodal Bias in Multimodal Deep Linear Networks

A Mathematical Framework to Characterize the Dependency Structures in Multimodal Learning with Minimax Principle

Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalization