Abstract:Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting , i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming—one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp) , a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt's context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.

PromptPose: Language Prompt Helps Animal Pose Estimation.

Animal Pose Estimation Based on Contrastive Learning with Dynamic Conditional Prompts

CLAMP: Prompt-based Contrastive Learning for Connecting Language and Animal Pose

Mutual Prompt Leaning for Vision Language Models

X-Pose: Detecting Any Keypoints

LAMP: Leveraging Language Prompts for Multi-person Pose Estimation

NLPrompt: Noise-Label Prompt Learning for Vision-Language Models

Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding

Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm

Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition

Revisiting Prompt Pretraining of Vision-Language Models

LoGoPrompt: Synthetic Text Images Can Be Good Visual Prompts for Vision-Language Models

Adaptive Multi-Modality Prompt Learning

Learning to Prompt for Vision-Language Models

VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generation

SuperAnimal pretrained pose estimation models for behavioral analysis

POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models

Supporting Vision-Language Model Inference with Confounder-pruning Knowledge Prompt

Rethinking Visual Prompt Learning as Masked Visual Token Modeling

DPL: Decoupled Prompt Learning for Vision-Language Models

Align and Prompt: Video-and-Language Pre-training with Entity Prompts