Abstract:With the growing success of multi-modal learning, research on the robustness of multi-modal models, especially when facing situations with missing modalities, is receiving increased attention. Nevertheless, previous studies in this domain exhibit certain limitations, as they often lack theoretical insights or their methodologies are tied to specific network architectures or modalities. We model the scenarios of multi-modal models encountering missing modalities from an information-theoretic perspective and illustrate that the performance ceiling in such scenarios can be approached by efficiently utilizing the information inherent in non-missing modalities. In practice, there are two key aspects: (1) The encoder should be able to extract sufficiently good features from the non-missing modality; (2) The extracted features should be robust enough not to be influenced by noise during the fusion process across modalities. To this end, we introduce Uni-Modal Ensemble with Missing Modality Adaptation (UME-MMA). UME-MMA employs uni-modal pre-trained weights for the multi-modal model to enhance feature extraction and utilizes missing modality data augmentation techniques to better adapt to situations with missing modalities. Apart from that, UME-MMA, built on a late-fusion learning framework, allows for the plug-and-play use of various encoders, making it suitable for a wide range of modalities and enabling seamless integration of large-scale pre-trained encoders to further enhance performance. And we demonstrate UME-MMA's effectiveness in audio-visual datasets (e.g., AV-MNIST, Kinetics-Sound, AVE) and vision-language datasets (e.g., MM-IMDB, UPMC Food101).

Improving Multi-Modal Learning with Uni-Modal Teachers

On Uni-Modal Feature Learning in Supervised Multi-Modal Learning

On Uni-modal Feature Learning in Multi-modal Learning

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

Cross-modality Online Distillation for Multi-View Action Recognition

Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

Improving the Modality Representation with Multi-View Contrastive Learning for Multimodal Sentiment Analysis

Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation

What Makes Multi-modal Learning Better than Single (Provably)

Learn to Combine Modalities in Multimodal Deep Learning

Improving Cross-Modal Image-Text Retrieval With Teacher-Student Learning

Improving Unimodal Inference with Multimodal Transformers

Multi-level Multi-task Representation Learning with Adaptive Fusion for Multimodal Sentiment Analysis

Cross-modality Representation Interactive Learning for Multimodal Sentiment Analysis

Uni-to-Multi Modal Knowledge Distillation for Bidirectional LiDAR-Camera Semantic Segmentation

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?

Enhanced Multimodal Representation Learning with Cross-modal KD.

Dense Multimodal Fusion for Hierarchically Joint Representation

Exploiting Multi-modal Fusion for Robust Face Representation Learning with Missing Modality

Multi-Modal Fusion-Based Multi-Task Semantic Communication System

One-stage Modality Distillation for Incomplete Multimodal Learning