Abstract:Multimodal representation learning is a challenging task in which previous work mostly focus on either uni-modality pre-training or cross-modality fusion. In fact, we regard modeling multimodal representation as building a skyscraper, where laying stable foundation and designing the main structure are equally essential. The former is like encoding robust uni-modal representation while the later is like integrating interactive information among different modalities, both of which are critical to learning an effective multimodal representation. Recently, contrastive learning has been successfully applied in representation learning, which can be utilized as the pillar of the skyscraper and benefit the model to extract the most important features contained in the multimodal data. In this paper, we propose a novel framework named MultiModal Contrastive Learning (MMCL) for multimodal representation to capture intra- and inter-modality dynamics simultaneously. Specifically, we devise uni-modal contrastive coding with an efficient uni-modal feature augmentation strategy to filter inherent noise contained in acoustic and visual modality and acquire more robust uni-modality representations. Besides, a pseudo siamese network is presented to predict representation across different modalities, which successfully captures cross-modal dynamics. Moreover, we design two contrastive learning tasks, instance- and sentiment-based contrastive learning, to promote the process of prediction and learn more interactive information related to sentiment. Extensive experiments conducted on two public datasets demonstrate that our method surpasses the state-of-the-art methods.

FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction

Multimodal Contrastive Training for Visual Representation Learning

Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis

Leveraging Vision-Language Pre-Trained Model and Contrastive Learning for Enhanced Multimodal Sentiment Analysis

CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection

Multi-Modal Representation via Contrastive Learning with Attention Bottleneck Fusion and Attentive Statistics Features

CLMLF: A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection

Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens

A multi-view contrastive learning for heterogeneous network embedding

Multimodal Representation Learning using Adaptive Graph Construction

Multimodal Pre-Training Based on Graph Attention Network for Document Understanding

CTHFNet: contrastive translation and hierarchical fusion network for text–video–audio sentiment analysis

Self-Supervised Intra-Modal and Cross-Modal Contrastive Learning for Point Cloud Understanding

On the Generalization of Multi-modal Contrastive Learning

Multimodal Sentiment Analysis Representations Learning via Contrastive Learning with Condense Attention Fusion

Text-Centric Multimodal Contrastive Learning for Sentiment Analysis

Contrastive Multimodal Fusion with TupleInfoNCE

GeoContrastNet: Contrastive Key-Value Edge Learning for Language-Agnostic Document Understanding

Contrastive Document Representation Learning with Graph Attention Networks

Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning.

Complementary multi-modality molecular self-supervised learning via non-overlapping masking for property prediction