Abstract:Prompt learning has become a prevalent strategy for adapting vision-language foundation models (VLMs) such as CLIP to downstream tasks. With the emergence of large language models (LLMs), recent studies have explored the potential of using category-related descriptions to enhance prompt effectiveness. However, conventional descriptions lack explicit structured information necessary to represent the interconnections among key elements like entities or attributes with relation to a particular category. Since existing prompt tuning methods give little consideration to managing structured knowledge, this paper advocates leveraging LLMs to construct a graph for each description to prioritize such structured knowledge. Consequently, we propose a novel approach called Hierarchical Prompt Tuning (HPT), enabling simultaneous modeling of both structured and conventional linguistic knowledge. Specifically, we introduce a relationship-guided attention module to capture pair-wise associations among entities and attributes for low-level prompt learning. In addition, by incorporating high-level and global-level prompts modeling overall semantics, the proposed hierarchical structure forges cross-level interlinks and empowers the model to handle more complex and long-term relationships. Finally, by enhancing multi-granularity knowledge generation, redesigning the relationship-driven attention re-weighting module, and incorporating consistent constraints on the hierarchical text encoder, we propose HPT++, which further improves the performance of HPT. Our experiments are conducted across a wide range of evaluation settings, including base-to-new generalization, cross-dataset evaluation, and domain generalization. Extensive results and ablation studies demonstrate the effectiveness of our methods, which consistently outperform existing SOTA methods.

MPT4LM: Multi-Modal Prompt Tuning Makes Pre-Trained Large Language Models Better Vision-Language Learners

M^2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

Progressive Multi-modal Conditional Prompt Tuning

Adapting Pre-trained Language Models to Vision-Language Tasksvia Dynamic Visual Prompting

Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting

Exploring the Transferability of Visual Prompting for Multimodal Large Language Models

MePT: Multi-Representation Guided Prompt Tuning for Vision-Language Model

MuDPT: Multi-modal Deep-symphysis Prompt Tuning for Large Pre-trained Vision-Language Models

Helping Language Models Learn More: Multi-dimensional Task Prompt for Few-shot Tuning

MixPrompt: Enhancing Generalizability and Adversarial Robustness for Vision-Language Models Via Prompt Fusion

Multitask Vision-Language Prompt Tuning

LAMM: Label Alignment for Multi-Modal Prompt Learning

Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment

HPT++: Hierarchically Prompting Vision-Language Models with Multi-Granularity Knowledge Generation and Improved Structure Modeling

Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models

Approximated Prompt Tuning for Vision-Language Pre-trained Models

MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting

Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models

Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning

Multi-modal Attribute Prompting for Vision-Language Models