Abstract:Pre-trained language models (PLMs) have played an increasing role in multimedia research. In terms of vision-language (VL) tasks, they often serve as a language encoder and still require an additional fusion network for VL reasoning, resulting in excessive memory overhead. In this paper, we focus on exploring PLMs as a stand-alone model for VL reasoning tasks. Inspired by the recently popular prompt tuning, we first prove that the processed visual features can be also projected onto the semantic space of PLMs and act as prompt tokens to bridge the gap between single- and multi-modal learning. However, this solution exhibits obvious redundancy in visual information and model inference, and the placement of prompt tokens also greatly affects the final performance. Based on these observations, we further propose a novel transfer learning approach for PLMs, termed Dynamic Visual Prompting (DVP). Concretely, DVP first deploys a cross-attention module to obtain text-related and compact visual prompt tokens, thereby greatly reducing the input length of PLMs. To obtain the optimal placement, we also equip DVP with a reinforcement-learning based search algorithm, which can automatically merge DVP with PLMs for different VL tasks via a very short search process. In addition, we also experiment DVP with the recently popular adapter approach to keep the most parameters of PLMs intact when adapting to VL tasks, helping PLMs achieve a quick shift between single- and multi-modal tasks. We apply DVP to two representative PLMs, namely BERT and T5, and conduct extensive experiments on a set of VL reasoning benchmarks including VQA2.0, GQA and SNLIVE. The experimental results not only show the advantage of DVP on efficiency and performance, but also confirm its superiority in adapting pre-trained language models to VL tasks.

Prompting Visual-Language Models for Efficient Video Understanding

ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization

Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs

Adapting Pre-trained Language Models to Vision-Language Tasksvia Dynamic Visual Prompting

Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting

Adaptively Building a Video-language Model for Video Captioning and Retrieval Without Massive Video Pretraining

Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization

Multi-Modal Prompting for Open-Vocabulary Video Visual Relationship Detection

PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models

Understanding Chinese Video and Language Via Contrastive Multimodal Pre-Training

Adaptive Spatio-Temporal Graph Enhanced Vision-Language Representation for Video QA

Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training

VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blending

Fine-Grained Visual Prompt Learning of Vision-Language Models for Image Recognition

EVLM: An Efficient Vision-Language Model for Visual Understanding

VoP: Text-Video Co-Operative Prompt Tuning for Cross-Modal Retrieval

LaViP:Language-Grounded Visual Prompts

Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning

PruneVid: Visual Token Pruning for Efficient Video Large Language Models

PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance

Leveraging Visual Prompts to Guide Language Modeling for Referring Video Object Segmentation