Abstract:Research on general multimodal systems has gained significant attention due to the proliferation of multimodal data in the real world. Despite the remarkable performance achieved by existing multimodal representation learning schemes, missing modalities remain a persistent issue, thereby limiting the overall applicability of multimodal systems. Intending to address the issue, we propose a novel approach named M 3 ixup (Multi-Modal Mixup), which leverages the mixup strategy to improve unimodal and multimodal representation learning while simultaneously increasing robustness against missing modalities. First, we adopt productive multimodal learning scheme to model representations with modality-specific and joint-modality encoders. The general scheme ensuring the proposed approach transferable for various multimodal learning scenarios, including supervised, unsupervised, and reinforcement learning. Then, the unimodal input and manifold mixup is used to enhance the modality-specific encoders to capture intra-modal dynamics. Next, we present multimodal mixup to mix different modalities and generate mixed multimodal representations in adapting and exploring steps. The former step aims at bridging the huge information gaps between unimodal and multimodal representations in the joint space in the alignment, while the latter step further captures the inter-modal dynamics and exploits the non-linear relationships among different modalities. After that, the mixed views are aligned with the original multimodal representations by contrastive learning. Additionally, we innovatively extend the mixup strategy to the loss function of multimodal contrastive learning in two steps to improve the alignment between mixed and original views. Extensive experiments on public datasets in various multimodal learning scenarios demonstrate the superiority of the proposed M 3 ixup. The codes are available at https://github.com/RH-Lin/m3ixup.

Enhancing cross-lingual transfer by mani-

Enhancing Cross-lingual Transfer by Manifold Mixup

Soul-Mix: Enhancing Multimodal Machine Translation with Manifold Mixup

Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation

Adaptive Token-level Cross-lingual Feature Mixing for Multilingual Neural Machine Translation

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning.

MSMix:An Interpolation-Based Text Data Augmentation Method Manifold Swap Mixup

XTransplant: A Probe into the Upper Bound Performance of Multilingual Capability and Culture Adaptability in LLMs via Mutual Cross-lingual Feed-forward Transplantation

Decoupled Mixup for Data-efficient Learning

AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging

SynerMix: Synergistic Mixup Solution for Enhanced Intra-Class Cohesion and Inter-Class Separability in Image Classification

Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment

Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast

Adapt and explore: Multimodal mixup for representation learning

Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral

Multi-Source Cross-Lingual Model Transfer: Learning What to Share

Boost Off/On-Manifold Adversarial Robustness for Deep Learning with Latent Representation Mixup.

Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models

PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment

STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation

mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?