Abstract:As a prominent parameter-efficient fine-tuning technique in NLP, prompt tuning is being explored its potential in computer vision. Typical methods for visual prompt tuning follow the sequential modeling paradigm stemming from NLP, which represents an input image as a flattened sequence of token embeddings and then learns a set of unordered parameterized tokens prefixed to the sequence representation as the visual prompts for task adaptation of large vision models. While such sequential modeling paradigm of visual prompt has shown great promise, there are two potential limitations. First, the learned visual prompts cannot model the underlying spatial relations in the input image, which is crucial for image encoding. Second, since all prompt tokens play the same role of prompting for all image tokens without distinction, it lacks the fine-grained prompting capability, i.e., individual prompting for different image tokens. In this work, we propose the \mymodel model (\emph{SA$^2$VP}), which learns a two-dimensional prompt token map with equal (or scaled) size to the image token map, thereby being able to spatially align with the image map. Each prompt token is designated to prompt knowledge only for the spatially corresponding image tokens. As a result, our model can conduct individual prompting for different image tokens in a fine-grained manner. Moreover, benefiting from the capability of preserving the spatial structure by the learned prompt token map, our \emph{SA$^2$VP} is able to model the spatial relations in the input image, leading to more effective prompting. Extensive experiments on three challenging benchmarks for image classification demonstrate the superiority of our model over other state-of-the-art methods for visual prompt tuning. Code is available at \emph{<a class="link-external link-https" href="https://github.com/tommy-xq/SA2VP" rel="external noopener nofollow">this https URL</a>}.

Fine-Grained Visual Text Prompting

Mutual Prompt Leaning for Vision Language Models

SA2VP: Spatially Aligned-and-Adapted Visual Prompt

Fine-Grained Visual Prompt Learning of Vision-Language Models for Image Recognition

Visual In-Context Prompting

TextRefiner: Internal Visual Feature as Efficient Refiner for Vision-Language Models Prompt Tuning

Explicit Visual Prompting for Universal Foreground Segmentations

SA$^2$VP: Spatially Aligned-and-Adapted Visual Prompt

TAI++: Text as Image for Multi-Label Image Classification by Co-Learning Transferable Prompt

Progressive Visual Prompt Learning with Contrastive Feature Re-formation

Towards Robust and Accurate Visual Prompting

Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection

GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation

Explicit Visual Prompting for Low-Level Structure Segmentations

MVP-SEG: Multi-View Prompt Learning for Open-Vocabulary Semantic Segmentation

VPA: Fully Test-Time Visual Prompt Adaptation

CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models

X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation

Progressive Multi-modal Conditional Prompt Tuning

EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model

Multi-modal Attribute Prompting for Vision-Language Models