Abstract:Research on general multimodal systems has gained significant attention due to the proliferation of multimodal data in the real world. Despite the remarkable performance achieved by existing multimodal representation learning schemes, missing modalities remain a persistent issue, thereby limiting the overall applicability of multimodal systems. Intending to address the issue, we propose a novel approach named M 3 ixup (Multi-Modal Mixup), which leverages the mixup strategy to improve unimodal and multimodal representation learning while simultaneously increasing robustness against missing modalities. First, we adopt productive multimodal learning scheme to model representations with modality-specific and joint-modality encoders. The general scheme ensuring the proposed approach transferable for various multimodal learning scenarios, including supervised, unsupervised, and reinforcement learning. Then, the unimodal input and manifold mixup is used to enhance the modality-specific encoders to capture intra-modal dynamics. Next, we present multimodal mixup to mix different modalities and generate mixed multimodal representations in adapting and exploring steps. The former step aims at bridging the huge information gaps between unimodal and multimodal representations in the joint space in the alignment, while the latter step further captures the inter-modal dynamics and exploits the non-linear relationships among different modalities. After that, the mixed views are aligned with the original multimodal representations by contrastive learning. Additionally, we innovatively extend the mixup strategy to the loss function of multimodal contrastive learning in two steps to improve the alignment between mixed and original views. Extensive experiments on public datasets in various multimodal learning scenarios demonstrate the superiority of the proposed M 3 ixup. The codes are available at https://github.com/RH-Lin/m3ixup.

Adapt and explore: Multimodal mixup for representation learning

Cultural Concept Adaptation on Multimodal Reasoning

Decoupled Mixup for Data-efficient Learning

Multimodal Representation Learning by Alternating Unimodal Adaptation

Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classification

Geodesic Multi-Modal Mixup for Robust Fine-Tuning

Harnessing Hard Mixed Samples with Decoupled Regularizer

MM-Mixing: Multi-Modal Mixing Alignment for 3D Understanding

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis

Un-mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning

Learn to Combine Modalities in Multimodal Deep Learning

MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance

Mixup Without Hesitation

Learning Unseen Modality Interaction

Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup

SynerMix: Synergistic Mixup Solution for Enhanced Intra-Class Cohesion and Inter-Class Separability in Image Classification

SUMix: Mixup with Semantic and Uncertain Information

Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup

Achieving Cross Modal Generalization with Multimodal Unified Representation.

MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts