Abstract:Multimodal representation learning is a challenging task in which previous work mostly focus on either uni-modality pre-training or cross-modality fusion. In fact, we regard modeling multimodal representation as building a skyscraper, where laying stable foundation and designing the main structure are equally essential. The former is like encoding robust uni-modal representation while the later is like integrating interactive information among different modalities, both of which are critical to learning an effective multimodal representation. Recently, contrastive learning has been successfully applied in representation learning, which can be utilized as the pillar of the skyscraper and benefit the model to extract the most important features contained in the multimodal data. In this paper, we propose a novel framework named MultiModal Contrastive Learning (MMCL) for multimodal representation to capture intra- and inter-modality dynamics simultaneously. Specifically, we devise uni-modal contrastive coding with an efficient uni-modal feature augmentation strategy to filter inherent noise contained in acoustic and visual modality and acquire more robust uni-modality representations. Besides, a pseudo siamese network is presented to predict representation across different modalities, which successfully captures cross-modal dynamics. Moreover, we design two contrastive learning tasks, instance- and sentiment-based contrastive learning, to promote the process of prediction and learn more interactive information related to sentiment. Extensive experiments conducted on two public datasets demonstrate that our method surpasses the state-of-the-art methods.

COM: Contrastive Masked-attention model for incomplete multimodal learning

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

MACO: A Modality Adversarial and Contrastive Framework for Modality-missing Multi-modal Knowledge Graph Completion

What to align in multimodal contrastive learning?

MECOM: A Meta-Completion Network for Fine-Grained Recognition With Incomplete Multi-Modalities

Multimodal Contrastive Training for Visual Representation Learning

Multimodal Adversarially Learned Inference with Factorized Discriminators

Incomplete Multimodal Learning for Remote Sensing Data Fusion

MAMO: Fine-Grained Vision-Language Representations Learning with Masked Multimodal Modeling

Learning Unseen Modality Interaction

Exemplar Masking for Multimodal Incremental Learning

Multimodal Contrastive Learning for Unpaired and Depth-privileged Semantic Segmentation.

ContrastMask: Contrastive Learning to Segment Every Thing

Learn to Combine Modalities in Multimodal Deep Learning

Self-Supervised Visual Representations Learning by Contrastive Mask Prediction

Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning

Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast

MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learning

Cross-modal Contrastive Learning for Multimodal Fake News Detection

On the Comparison between Multi-modal and Single-modal Contrastive Learning

Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis