Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning

Zhengqing Gao,Xiang Ao,Xu-Yao Zhang,Cheng-Lin Liu

2024-08-29

Abstract:Adapting pre-trained models to open classes is a challenging problem in machine learning. Vision-language models fully explore the knowledge of text modality, demonstrating strong zero-shot recognition performance, which is naturally suited for various open-set problems. More recently, some research focuses on fine-tuning such models to downstream tasks. Prompt tuning methods achieved huge improvements by learning context vectors on few-shot data. However, through the evaluation under open-set adaptation setting with the test data including new classes, we find that there exists a dilemma that learned prompts have worse generalization abilities than hand-crafted prompts. In this paper, we consider combining the advantages of both and come up with a test-time prompt tuning approach, which leverages the maximum concept matching (MCM) scores as dynamic weights to generate an input-conditioned prompt for each image during test. Through extensive experiments on 11 different datasets, we show that our proposed method outperforms all comparison methods on average considering both base and new classes. The code is available at <a class="link-external link-https" href="https://github.com/gaozhengqing/TTPT" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily focuses on improving the performance of Vision-Language Models (VLMs) in open class scenarios. Specifically:

1. **Open Class Adaptation Problem**: The paper points out that in real-world applications, pre-trained models encounter various out-of-distribution (OOD) data. Although VLMs perform well in zero-shot classification tasks, they still face challenges in open class adaptation settings.

2. **Issues with Existing Methods**:
- **CoOp**: Optimizes the model by learning context vectors but tends to overfit to base classes, resulting in poor generalization to new classes.
- **CoCoOp**: Uses a lightweight neural network to generate input-dependent context vectors to alleviate the overfitting issue but sacrifices some accuracy on base classes.

3. **Proposed Method**: The paper proposes a test-time prompt tuning method that combines the advantages of hand-crafted prompts and learned prompts. Specifically, this method uses the Maximum Concept Matching (MCM) score as dynamic weights to generate input-conditioned prompts, thereby improving performance on both base and new classes.

Experiments on multiple datasets validate that this method outperforms existing methods in open class adaptation tasks.