Abstract:Cross-modal alignment plays a crucial role in vision-language pre-training (VLP) models, enabling them to capture meaningful associations across different modalities. For this purpose, numerous masked modeling tasks have been proposed for VLP to further promote cross-modal interactions. The core idea of previous masked modeling tasks is to focus on reconstructing the masked tokens based on visible context for learning local-local alignment. However, most of them pay little attention to the global semantic features generated for the masked data, resulting in a limited cross-modal alignment ability of global representations to local features of the other modality. Therefore, in this paper, we propose a novel Global and Local Semantic Completion Learning (GLSCL) task to facilitate global-local alignment and local-local alignment simultaneously. Specifically, the GLSCL task complements the missing semantics of masked data and recovers global and local features by cross-modal interactions. Our GLSCL consists of masked global semantic completion (MGSC) and masked local token completion (MLTC). MGSC promotes learning more representative global features, which have a great impact on the performance of downstream tasks, while MLTC reconstructs modal-fusion local tokens, further enhancing accurate comprehension of multimodal data. To evaluate the proposed approaches on cross-modal alignment, we develop a validation benchmark called ALIGN-BENCH. Moreover, we present a flexible vision encoder, enabling our model to simultaneously perform image-text and video-text multimodal tasks. Experimental results show that our proposed method obtains state-of-the-art performance on various vision-language benchmarks, such as visual question answering, image-text retrieval, and video-text retrieval.

Superpixel Semantics Representation and Pre-training for Vision-Language Task

Semantic Communication Approach for Multi-Task Image Transmission

A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model

3D Vision-Language Gaussian Splatting

Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learning.

Vision-language pre-training via modal interaction

Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends

Global and Local Semantic Completion Learning for Vision-Language Pre-training

Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation

Building a Strong Pre-Training Baseline for Universal 3D Large-Scale Perception

Semantic-Aware Fusion Network Based on Super-Resolution

SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidance

SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks

Leveraging per Image-Token Consistency for Vision-Language Pre-training

End-to-End Unsupervised Vision-and-Language Pre-training with Referring Expression Matching.

3D Scene Graph Guided Vision-Language Pre-training

12-in-1: Multi-Task Vision and Language Representation Learning

SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels

Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding

MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning

Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts