Abstract:Vision-Language Pre-training (VLP) aims to learn multi-modal representations from image-text pairs and serves for downstream vision-language tasks in a finetuning fashion. The dominant VLP models adopt a CNN-Transformer architecture, which embeds images with a CNN, and then aligns images and texts with a Transformer. Visual relationship between visual contents plays an important role in image understanding and is crucial for inter-modal alignment learning in VLP. However, CNNs have limitations in visual relation learning due to local receptive field's weakness in modeling long-range dependencies. Thus the two objectives of learning visual relation and inter-modal alignment are encapsulated in the same Transformer network. Such design might restrict the inter-modal alignment learning in the Transformer by neglecting the specialized characteristic of each objective. To tackle this challenge, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment. Specifically, we propose a metric named Inter-Modality Flow (IMF) to measure the interaction between vision and language (i.e., inter-modality). We also design a novel masking optimization mechanism named Masked Feature Regression (MFR) in Transformer to further promote the inter-modality learning. To the best of our knowledge, this is the first study to explore the benefit of Transformer for visual feature learning in VLP. We verify our method on a wide range of vision-language tasks, including Image-Text Retrieval, Visual Question Answering (VQA), Visual Entailment and Visual Reasoning. The result shows that our approach not only outperforms the state-of-the-art VLP models, but also exhibits superiority on the IMF metric.

End-to-End Unsupervised Vision-and-Language Pre-training with Referring Expression Matching.

VLP: A Survey on Vision-language Pre-training

SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels

Unified Vision-Language Pre-Training for Image Captioning and VQA

Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models

Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learning.

Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training

Vision-language pre-training via modal interaction

Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends

Generalizing Multimodal Pre-training into Multilingual via Language Acquisition

Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages

VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

3D Vision and Language Pretraining with Large-Scale Synthetic Data

UNIMO-2: End-to-End Unified Vision-Language Grounded Learning

Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Unsupervised Domain Adaption Harnessing Vision-Language Pre-training

Fine-Grained Semantically Aligned Vision-Language Pre-Training

VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix