Visual Prompting in Multimodal Large Language Models: A Survey

Junda Wu,Zhehao Zhang,Yu Xia,Xintong Li,Zhaoyang Xia,Aaron Chang,Tong Yu,Sungchul Kim,Ryan A. Rossi,Ruiyi Zhang,Subrata Mitra,Dimitris N. Metaxas,Lina Yao,Jingbo Shang,Julian McAuley

2024-09-05

Abstract:Multimodal large language models (MLLMs) equip pre-trained large-language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-grained and free-form visual instructions. This paper presents the first comprehensive survey on visual prompting methods in MLLMs, focusing on visual prompting, prompt generation, compositional reasoning, and prompt learning. We categorize existing visual prompts and discuss generative methods for automatic prompt annotations on the images. We also examine visual prompting methods that enable better alignment between visual encoders and backbone LLMs, concerning MLLM's visual grounding, object referring, and compositional reasoning abilities. In addition, we provide a summary of model training and in-context learning methods to improve MLLM's perception and understanding of visual prompts. This paper examines visual prompting methods developed in MLLMs and provides a vision of the future of these methods.

Machine Learning,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily aims to address the limitations of Multimodal Large Language Models (MLLMs) in processing visual information. Specifically:

1. **Visual Alignment Issue**: Traditional text prompt methods are inadequate in describing and specifying visual elements, leading to inaccurate visual alignment and the inability to reference visual information in detail. This can cause visual hallucinations and language bias.

2. **Fine-Grained Instruction Issue**: With the advent of visual prompt methods, they can provide more fine-grained (pixel-level) multimodal input instructions, making general prompt templates unsuitable for different images. Therefore, there is a need to generate instance-level visual prompts.

3. **Model Training and Fine-Tuning Issue**: Although visual prompt methods have enhanced the visual capabilities of MLLMs, these models may still misalign with visual prompts due to the lack of diverse visual prompt data during the pre-training phase. Additionally, the paper explores how pre-training, fine-tuning, and contextual learning methods can better align MLLMs with enhanced multimodal prompts.

In summary, this paper aims to improve the visual understanding and reasoning capabilities of MLLMs in handling complex multimodal tasks by comprehensively categorizing existing visual prompt methods and proposing improvements.