Abstract:PDF HTML XML Export Cite reminder Multimodal Pre-training Method for Vision-language Understanding and Generation DOI: 10.21655/ijsi.1673-7288.00315 Author: Affiliation: Clc Number: Fund Project: Article | Figures | Metrics | Reference | Related | Cited by | Materials | Comments Abstract:Most existing vision-language pre-training methods focus on understanding tasks and use BERT-like loss functions (masked language modeling and image-text matching) during pre-training. Despite their good performance in the understanding of downstream tasks, such as visual question answering, image-text retrieval, and visual entailment, these methods cannot generate information. To tackle this problem, this study proposes Unified multimodal pre-training for Vision-Language understanding and generation (UniVL). The proposed UniVL is capable of handling both understanding tasks and generation tasks. It expands existing pre-training paradigms and uses random masks and causal masks simultaneously, where causal masks are triangular masks that mask future tokens, and such pre-trained models can have autoregressive generation abilities. Moreover, several vision-language understanding tasks are turned into text generation tasks according to specifications, and the prompt-based method is employed for fine-tuning of different downstream tasks. The experiments show that there is a trade-off between understanding tasks and generation tasks when the same model is used, and a feasible way to improve both tasks is to use more data. The proposed UniVL framework attains comparable performance to recent vision-language pre-training methods in both understanding tasks and generation tasks. Moreover, the prompt-based generation method is more effective and even outperforms discriminative methods in few-shot scenarios. Reference Related Cited by

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training

Unified Vision-Language Pre-Training for Image Captioning and VQA

UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

UNITER: UNiversal Image-TExt Representation Learning

UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

Multi-modal Universal Embedding Representations for Language Understanding

UNIMO-2: End-to-End Unified Vision-Language Grounded Learning

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

UniCode: Learning a Unified Codebook for Multimodal Large Language Models

OmniVL: One Foundation Model for Image-Language and Video-Language Tasks

Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

OmniVL:One Foundation Model for Image-Language and Video-Language Tasks

VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation

X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

Multimodal Pre-training Method for Vision-language Understanding and Generation.

From Unimodal to Multimodal: Scaling up Projectors to Align Modalities

VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation

Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval