Abstract:Multi-modal Contrastive Representation learning aims to encode different modalities into a semantically aligned shared space. This paradigm shows remarkable generalization ability on numerous downstream tasks across various modalities. However, the reliance on massive high-quality data pairs limits its further development on more modalities. This paper proposes a novel training-efficient method for learning MCR without paired data called Connecting Multi-modal Contrastive Representations (C-MCR). Specifically, given two existing MCRs pre-trained on (A, B) and (B, C) modality pairs, we project them to a new space and use the data from the overlapping modality B to aligning the two MCRs in the new space. Meanwhile, since the modality pairs (A, B) and (B, C) are already aligned within each MCR, the connection learned by overlapping modality can also be transferred to non-overlapping modality pair (A, C). To unleash the potential of C-MCR, we further introduce a semantic-enhanced inter- and intra-MCR connection method. We first enhance the semantic consistency and completion of embeddings across different modalities for more robust alignment. Then we utilize the inter-MCR alignment to establish the connection, and employ the intra-MCR alignment to better maintain the connection for inputs from non-overlapping modalities. To demonstrate the effectiveness of C-MCR, we connect CLIP and CLAP via texts to derive audio-visual representations, and integrate CLIP and ULIP via images for 3D-language representations. Remarkably, without using any paired data, C-MCR for audio-visual achieves state-of-the-art performance on audio-image retrieval, audio-visual source localization, and counterfactual audio-image recognition tasks. Furthermore, C-MCR for 3D-language also attains advanced zero-shot 3D point cloud classification accuracy on ModelNet40.

Contrastive Learning between Classical and Modern Chinese for Classical Chinese Machine Reading Comprehension

Multi-Matching Network for Multiple Choice Reading Comprehension

Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond

XCMRC: Evaluating Cross-lingual Machine Reading Comprehension

Chinese Machine Reading Comprehension Based on Language Model Containing Knowledge.

Feeding What You Need by Understanding What You Learned

On the Comparison between Multi-modal and Single-modal Contrastive Learning

A cross-temporal contrastive disentangled model for ancient Chinese understanding

A Sentence Cloze Dataset for Chinese Machine Reading Comprehension

Learn a prior question-aware feature for machine reading comprehension

Combining permuted language model and adversarial training for Chinese machine reading comprehension

CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models

From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Model to Pre-trained Machine Reader

Text-Centric Multimodal Contrastive Learning for Sentiment Analysis

From Good to Best: Two-Stage Training for Cross-Lingual Machine Reading Comprehension

DCMN+: Dual Co-Matching Network for Multi-choice Reading Comprehension

A machine reading comprehension model with counterfactual contrastive learning for emotion-cause pair extraction

Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding

Option Comparison Network for Multiple-choice Reading Comprehension

Connecting Multi-modal Contrastive Representations

Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis