Abstract:Cross-modal alignment plays a crucial role in vision-language pre-training (VLP) models, enabling them to capture meaningful associations across different modalities. For this purpose, numerous masked modeling tasks have been proposed for VLP to further promote cross-modal interactions. The core idea of previous masked modeling tasks is to focus on reconstructing the masked tokens based on visible context for learning local-local alignment. However, most of them pay little attention to the global semantic features generated for the masked data, resulting in a limited cross-modal alignment ability of global representations to local features of the other modality. Therefore, in this paper, we propose a novel Global and Local Semantic Completion Learning (GLSCL) task to facilitate global-local alignment and local-local alignment simultaneously. Specifically, the GLSCL task complements the missing semantics of masked data and recovers global and local features by cross-modal interactions. Our GLSCL consists of masked global semantic completion (MGSC) and masked local token completion (MLTC). MGSC promotes learning more representative global features, which have a great impact on the performance of downstream tasks, while MLTC reconstructs modal-fusion local tokens, further enhancing accurate comprehension of multimodal data. To evaluate the proposed approaches on cross-modal alignment, we develop a validation benchmark called ALIGN-BENCH. Moreover, we present a flexible vision encoder, enabling our model to simultaneously perform image-text and video-text multimodal tasks. Experimental results show that our proposed method obtains state-of-the-art performance on various vision-language benchmarks, such as visual question answering, image-text retrieval, and video-text retrieval.

MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning

MVPTR: Multi-Stage Vision-Language Pre-Training via Multi-Level Semantic Alignment

Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

Vision-language pre-training via modal interaction

Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training

Global and Local Semantic Completion Learning for Vision-Language Pre-training

MVP: Multimodality-Guided Visual Pre-training

Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learning.

Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Multimodal Pretraining from Monolingual to Multilingual

SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels

Leveraging per Image-Token Consistency for Vision-Language Pre-training

Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training

Refined Vision-Language Modeling for Fine-grained Multi-modal Pre-training

Fine-Grained Semantically Aligned Vision-Language Pre-Training

Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?

Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training

VL-Meta: Vision-Language Models for Multimodal Meta-Learning

Vision-Language Pre-Training with Triple Contrastive Learning