Abstract:Recent Vision-Language Pretrained (VLP) models have become the backbone for many downstream tasks, but they are utilized as frozen model without learning. Prompt learning is a method to improve the pre-trained VLP model by adding a learnable context vector to the inputs of the text encoder. In a few-shot learning scenario of the downstream task, MLE training can lead the context vector to over-fit dominant image features in the training data. This overfitting can potentially harm the generalization ability, especially in the presence of a distribution shift between the training and test dataset. This paper presents a Bayesian-based framework of prompt learning, which could alleviate the overfitting issues on few-shot learning application and increase the adaptability of prompts on unseen instances. Specifically, modeling data-dependent prior enhances the adaptability of text features for both seen and unseen image features without the trade-off of performance between them. Based on the Bayesian framework, we utilize the Wasserstein Gradient Flow in the estimation of our target posterior distribution, which enables our prompt to be flexible in capturing the complex modes of image features. We demonstrate the effectiveness of our method on benchmark datasets for several experiments by showing statistically significant improvements on performance compared to existing methods. The code is available at

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the insufficient generalization ability of existing vision - language pre - training models (VLP) in downstream tasks in the few - shot learning scenario. Specifically: 1. **Over - fitting problem**: In the few - shot learning scenario, maximum likelihood estimation (MLE) training may cause the context vector to over - fit the dominant image features in the training data, thus harming the generalization ability of the model, especially when there are distribution differences between the training set and the test set. 2. **Insufficient adaptability**: Traditional prompt learning methods are difficult to capture multi - modal image features and perform poorly when dealing with unseen data instances, resulting in performance degradation. To solve these problems, the paper proposes Adaptive Particle - based Prompt Learning (APP), which is a prompt learning framework based on Bayesian inference. By introducing a data - dependent prior, APP can enhance the adaptability of prompts to seen and unseen instances and improve the flexibility of the model on complex image features. In addition, the paper also uses the Wasserstein gradient flow to approximate the posterior distribution, enabling prompts to more flexibly capture the complex patterns of image features. ### Specific improvement points: - **Enhancing the flexibility of prompts**: By using the Wasserstein gradient flow to approximate the prompt posterior distribution, the context vector can more flexibly infer the complex image feature space. - **Enhancing the adaptability of prompts**: By modeling the data - prior distribution that depends on image feature information, the text features can capture the multi - modality of seen image features and adapt to unseen image features, thereby improving the performance of unseen data without sacrificing the performance of seen data. In conclusion, this paper aims to alleviate the over - fitting problem in few - shot learning and improve the generalization ability of the model under different data distributions through the prompt learning method in the Bayesian framework.

Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior

Bayesian Prompt Learning for Image-Language Model Generalization

Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models

Differentiable Prompt Learning for Vision Language Models

Active Prompt Learning with Vision-Language Model Priors

Prompt Distribution Learning

Revisiting Prompt Pretraining of Vision-Language Models

PromptCD: Coupled and Decoupled Prompt Learning for Vision-Language Models

MixPrompt: Enhancing Generalizability and Adversarial Robustness for Vision-Language Models Via Prompt Fusion

Consistency-guided Prompt Learning for Vision-Language Models

Mutual Prompt Leaning for Vision Language Models

Learning to Prompt with Text Only Supervision for Vision-Language Models

Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models

Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model

BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction

NLPrompt: Noise-Label Prompt Learning for Vision-Language Models

DPL: Decoupled Prompt Learning for Vision-Language Models

LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models

MoPD: Mixture-of-Prompts Distillation for Vision-Language Models

Probabilistic Prompt Learning for Dense Prediction

PRE: Vision-Language Prompt Learning with Reparameterization Encoder