Abstract:With the flourishing of social media platforms, vision-language pre-training (VLP) recently has received great attention and many remarkable progresses have been achieved. The success of VLP largely benefits from the information complementation and enhancement between different modalities. However, most of recent studies focus on cross-modal contrastive learning (CMCL) to promote image-text alignment by pulling embeddings of positive sample pairs together while pushing those of negative pairs apart, which ignores the natural asymmetry property between different modalities and requires large-scale image-text corpus to achieve arduous progress. To mitigate this predicament, we propose CMAL, a Cross-Modal Associative Learning framework with anchor points detection and cross-modal associative learning for VLP. Specifically, we first respectively embed visual objects and textual tokens into separate hypersphere spaces to learn intra-modal hidden features, and then design a cross-modal associative prompt layer to perform anchor point masking and swap feature filling for constructing a hybrid cross-modal associative prompt. Afterwards, we exploit a unified semantic encoder to learn their cross-modal interactive features for context adaptation. Finally, we design an associative mapping classification layer to learn potential associative mappings between modalities at anchor points, within which we develop a fresh self-supervised associative mapping classification task to boost CMAL's performance. Experimental results verify the effectiveness of CMAL, showing that it achieves competitive performance against previous CMCL-based methods on four common downstream vision-and-language tasks, with significantly fewer corpus. Especially, CMAL obtains new state-of-the-art results on SNLI-VE and REC (testA).

RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-training.

Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

Dense Contrastive Visual-Linguistic Pretraining

UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

Multimodal Pretraining from Monolingual to Multilingual

Vision-Language Pre-Training with Triple Contrastive Learning

Contrastive Visual-Linguistic Pretraining

VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix

Cross-modality interaction reasoning for enhancing vision-language pre-training in image-text retrieval

CAVL: Learning Contrastive and Adaptive Representations of Vision and Language

Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training

WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training

CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training

WenLan: Efficient Large-Scale Multi-Modal Pre-Training on Real World Data

Multimodal Contrastive Training for Visual Representation Learning

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classification

Leveraging per Image-Token Consistency for Vision-Language Pre-training

Vision-language pre-training via modal interaction

ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training

Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation