Abstract:As a prominent parameter-efficient fine-tuning technique in NLP, prompt tuning is being explored its potential in computer vision. Typical methods for visual prompt tuning follow the sequential modeling paradigm stemming from NLP, which represents an input image as a flattened sequence of token embeddings and then learns a set of unordered parameterized tokens prefixed to the sequence representation as the visual prompts for task adaptation of large vision models. While such sequential modeling paradigm of visual prompt has shown great promise, there are two potential limitations. First, the learned visual prompts cannot model the underlying spatial relations in the input image, which is crucial for image encoding. Second, since all prompt tokens play the same role of prompting for all image tokens without distinction, it lacks the fine-grained prompting capability, i.e., individual prompting for different image tokens. In this work, we propose the Spatially Aligned-and-Adapted Visual Prompt model (SA^2VP), which learns a two-dimensional prompt token map with equal (or scaled) size to the image token map, thereby being able to spatially align with the image map. Each prompt token is designated to prompt knowledge only for the spatially corresponding image tokens. As a result, our model can conduct individual prompting for different image tokens in a fine-grained manner. Moreover, benefiting from the capability of preserving the spatial structure by the learned prompt token map, our SA^2VP is able to model the spatial relations in the input image, leading to more effective prompting. Extensive experiments on three challenging benchmarks for image classification demonstrate the superiority of our model over other state-of-the-art methods for visual prompt tuning. Code is available at https://github.com/tommy-xq/SA2VP.

Rethinking Remote Sensing Pretrained Model: Instance-Aware Visual Prompting for Remote Sensing Scene Classification.

MVP: Meta Visual Prompt Tuning for Few-Shot Remote Sensing Image Scene Classification.

An Empirical Study of Remote Sensing Pretraining

Visual Prompt Tuning

E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning

Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?

Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models

Advancing Plain Vision Transformer Toward Remote Sensing Foundation Model

LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning

IV-tuning: Parameter-Efficient Transfer Learning for Infrared-Visible Tasks

Remote Sensing Scene Classification Based on Local Selection Vision Transformer

PVP: Pre-trained Visual Parameter-Efficient Tuning

Pro-tuning: Unified Prompt Tuning for Vision Tasks

VPPT: Visual Pre-Trained Prompt Tuning Framework for Few-Shot Image Classification

Improving Visual Prompt Tuning for Self-supervised Vision Transformers

PERS: Parameter-Efficient Multi-modal Transfer Learning for Remote Sensing Visual Question Answering

SA2VP: Spatially Aligned-and-Adapted Visual Prompt

Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrieval

PERS: Parameter-Efficient Multimodal Transfer Learning for Remote Sensing Visual Question Answering

Unleashing the Power of Visual Prompting At the Pixel Level