Abstract:Vision-Language Models (VLMs), such as CLIP, have demonstrated impressive zero-shot transfer capabilities in image-level visual perception. However, these models have shown limited performance in instance-level tasks that demand precise localization and recognition. Previous works have suggested that incorporating visual prompts, such as colorful boxes or circles, can improve the ability of models to recognize objects of interest. Nonetheless, compared to language prompting, visual prompting designs are rarely explored. Existing approaches, which employ coarse visual cues such as colorful boxes or circles, often result in sub-optimal performance due to the inclusion of irrelevant and noisy pixels. In this paper, we carefully study the visual prompting designs by exploring more fine-grained markings, such as segmentation masks and their variations. In addition, we introduce a new zero-shot framework that leverages pixel-level annotations acquired from a generalist segmentation model for fine-grained visual prompting. Consequently, our investigation reveals that a straightforward application of blur outside the target mask, referred to as the Blur Reverse Mask, exhibits exceptional effectiveness. This proposed prompting strategy leverages the precise mask annotations to reduce focus on weakly related regions while retaining spatial coherence between the target and the surrounding background. Our **F**ine-**G**rained **V**isual **P**rompting (**FGVP**) demonstrates superior performance in zero-shot comprehension of referring expressions on the RefCOCO, RefCOCO+, and RefCOCOg benchmarks. It outperforms prior methods by an average margin of 3.0\% to 4.6\%, with a maximum improvement of 12.5\% on the RefCOCO+ testA subset. The part detection experiments conducted on the PACO dataset further validate the preponderance of FGVP over existing visual prompting techniques. Code is available at https://github.com/ylingfeng/FGVP.

Progressive Visual Prompt Learning with Contrastive Feature Re-formation

Revisiting Prompt Pretraining of Vision-Language Models

Progressive Multi-modal Conditional Prompt Tuning

Unleashing the Power of Visual Prompting At the Pixel Level

Mutual Prompt Leaning for Vision Language Models

Towards Robust and Accurate Visual Prompting

Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting

Fine-Grained Visual Prompting

Adapting Pre-trained Language Models to Vision-Language Tasksvia Dynamic Visual Prompting

Rethinking Visual Prompt Learning as Masked Visual Token Modeling

MixPrompt: Enhancing Generalizability and Adversarial Robustness for Vision-Language Models Via Prompt Fusion

SA$^2$VP: Spatially Aligned-and-Adapted Visual Prompt

Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models

SA2VP: Spatially Aligned-and-Adapted Visual Prompt

Prompt-aligned Gradient for Prompt Tuning

AdaViPro: Region-based Adaptive Visual Prompt for Large-Scale Models Adapting

VTPL: Visual and Text Prompt Learning for Visual-Language Models

LoGoPrompt: Synthetic Text Images Can Be Good Visual Prompts for Vision-Language Models

Consistency-guided Prompt Learning for Vision-Language Models

VPA: Fully Test-Time Visual Prompt Adaptation

InsVP: Efficient Instance Visual Prompting from Image Itself