Abstract:With the flourishing of social media platforms, vision-language pre-training (VLP) recently has received great attention and many remarkable progresses have been achieved. The success of VLP largely benefits from the information complementation and enhancement between different modalities. However, most of recent studies focus on cross-modal contrastive learning (CMCL) to promote image-text alignment by pulling embeddings of positive sample pairs together while pushing those of negative pairs apart, which ignores the natural asymmetry property between different modalities and requires large-scale image-text corpus to achieve arduous progress. To mitigate this predicament, we propose CMAL, a Cross-Modal Associative Learning framework with anchor points detection and cross-modal associative learning for VLP. Specifically, we first respectively embed visual objects and textual tokens into separate hypersphere spaces to learn intra-modal hidden features, and then design a cross-modal associative prompt layer to perform anchor point masking and swap feature filling for constructing a hybrid cross-modal associative prompt. Afterwards, we exploit a unified semantic encoder to learn their cross-modal interactive features for context adaptation. Finally, we design an associative mapping classification layer to learn potential associative mappings between modalities at anchor points, within which we develop a fresh self-supervised associative mapping classification task to boost CMAL's performance. Experimental results verify the effectiveness of CMAL, showing that it achieves competitive performance against previous CMCL-based methods on four common downstream vision-and-language tasks, with significantly fewer corpus. Especially, CMAL obtains new state-of-the-art results on SNLI-VE and REC (testA).

ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross-and Intra-modal Knowledge Integration

ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration

Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learning.

Harnessing Vision-Language Pretrained Models with Temporal-Aware Adaptation for Referring Video Object Segmentation

Refined Vision-Language Modeling for Fine-grained Multi-modal Pre-training

Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?

Cross-modality interaction reasoning for enhancing vision-language pre-training in image-text retrieval

Global and Local Semantic Completion Learning for Vision-Language Pre-training

Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects

Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training

SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels

Probing Cross-modal Semantics Alignment Capability from the Textual Perspective

Unified Lexical Representation for Interpretable Visual-Language Alignment

Vision-and-Language Navigation via Latent Semantic Alignment Learning

Fine-Grained Semantically Aligned Vision-Language Pre-Training

CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training

Leveraging per Image-Token Consistency for Vision-Language Pre-training

Retrieval-based Knowledge Augmented Vision Language Pre-training

Vision-language pre-training via modal interaction

Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment