Abstract:Vision-Language Pre-training (VLP) aims to learn multi-modal representations from image-text pairs and serves for downstream vision-language tasks in a finetuning fashion. The dominant VLP models adopt a CNN-Transformer architecture, which embeds images with a CNN, and then aligns images and texts with a Transformer. Visual relationship between visual contents plays an important role in image understanding and is crucial for inter-modal alignment learning in VLP. However, CNNs have limitations in visual relation learning due to local receptive field's weakness in modeling long-range dependencies. Thus the two objectives of learning visual relation and inter-modal alignment are encapsulated in the same Transformer network. Such design might restrict the inter-modal alignment learning in the Transformer by neglecting the specialized characteristic of each objective. To tackle this challenge, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment. Specifically, we propose a metric named Inter-Modality Flow (IMF) to measure the interaction between vision and language (i.e., inter-modality). We also design a novel masking optimization mechanism named Masked Feature Regression (MFR) in Transformer to further promote the inter-modality learning. To the best of our knowledge, this is the first study to explore the benefit of Transformer for visual feature learning in VLP. We verify our method on a wide range of vision-language tasks, including Image-Text Retrieval, Visual Question Answering (VQA), Visual Entailment and Visual Reasoning. The result shows that our approach not only outperforms the state-of-the-art VLP models, but also exhibits superiority on the IMF metric.

Enhancing Sentence Representation with Visually-supervised Multimodal Pre-training

VLP2MSA: Expanding Vision-Language Pre-Training to Multimodal Sentiment Analysis

Multimodal Pretraining from Monolingual to Multilingual

Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

ViT-Lens: Towards Omni-modal Representations

Semi-supervised Visual Feature Integration for Language Models Through Sentence Visualization

Vision-language pre-training via modal interaction

Leveraging Vision-Language Pre-Trained Model and Contrastive Learning for Enhanced Multimodal Sentiment Analysis

UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

Visually-Augmented Language Modeling

Universal Multimodal Representation for Language Understanding

A Survey of Vision-Language Pre-Trained Models

Semi-supervised Visual Feature Integration for Pre-trained Language Models

Sentiment-aware Multimodal Pre-Training for Multimodal Sentiment Analysis

Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training

Probing Contextualized Sentence Representations with Visual Awareness

End-to-End Unsupervised Vision-and-Language Pre-training with Referring Expression Matching.

Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization

Unified Video-Language Pre-training with Synchronized Audio