Abstract:The emergence of large pre-trained vision-language models (VLMs) represents a paradigm shift in machine learning, with unprecedented results in a broad span of visual recognition tasks. CLIP, one of the most popular VLMs, has exhibited remarkable zero-shot and transfer learning capabilities in classification. To transfer CLIP to downstream tasks, adapters constitute a parameter-efficient approach that avoids backpropagation through the large model (unlike related prompt learning methods). However, CLIP adapters have been developed to target discriminative performance, and the quality of their uncertainty estimates has been overlooked. In this work we show that the discriminative performance of state-of-the-art CLIP adapters does not always correlate with their uncertainty estimation capabilities, which are essential for a safe deployment in real-world scenarios. We also demonstrate that one of such adapters is obtained through MAP inference from a more general probabilistic framework. Based on this observation we introduce BayesAdapter, which leverages Bayesian inference to estimate a full probability distribution instead of a single point, better capturing the variability inherent in the parameter space. In a comprehensive empirical evaluation we show that our approach obtains high quality uncertainty estimates in the predictions, standing out in calibration and selective classification. Our code is publicly available at: <a class="link-external link-https" href="https://github.com/pablomorales92/BayesAdapter" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the deficiency of existing CLIP adapters in uncertainty estimation. Although the existing CLIP adapters perform excellently in discriminative performance (such as classification accuracy), the quality of their uncertainty estimation has been ignored. This is a serious problem in practical applications, because good uncertainty estimation is crucial for ensuring the safe deployment of the model in real - world scenarios. Specifically, the paper points out: 1. **Inconsistency between discriminative performance and uncertainty estimation**: Some methods that perform best in test accuracy perform poorly in other metrics involving confidence scores (such as calibration and coverage under high confidence). 2. **Limitations of existing methods**: For example, the CLAP method is inferred by maximum a posteriori probability (MAP), which means it only depends on a point estimate of the parameters, resulting in less - than - ideal handling of uncertainty. 3. **The need to introduce Bayesian inference**: In order to better handle uncertainty, the authors propose using Bayesian inference to estimate the full probability distribution of the parameters instead of a single point estimate. Therefore, the main goal of this paper is to improve the uncertainty estimation ability of CLIP adapters by introducing a new method - BayesAdapter, which uses Bayesian inference, while maintaining the competitiveness of its discriminative performance. ### Key contributions of BayesAdapter - **Analyzing the uncertainty estimation quality of existing CLIP adapters**: Revealing the lack of correlation between the discriminative performance of current adapters and their uncertainty estimation ability. - **Demonstrating the limitations of the CLAP method**: Proving that the CLAP method is actually a maximum a posteriori probability (MAP) estimate and can only provide a point estimate of the parameters. - **Introducing BayesAdapter**: By Bayesian inference, estimating the probability distribution of the parameters, thereby achieving more abundant uncertainty quantification. - **Experimental proof of superiority**: Experiments on multiple datasets show that BayesAdapter significantly outperforms existing deterministic adapters in tasks related to uncertainty estimation (such as calibration and high - confidence selective classification), while remaining competitive in discriminative performance. In conclusion, this paper aims to solve the deficiency of CLIP adapters in uncertainty estimation and proposes a new method based on Bayesian inference to improve the reliability and safety of the model in practical applications.

BayesAdapter: enhanced uncertainty estimation in CLIP few-shot adaptation

BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning

ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models

Bayesian Exploration of Pre-trained Models for Low-shot Image Classification

Robust Calibration of Large Vision-Language Adapters

Proto-Adapter: Efficient Training-Free CLIP-Adapter for Few-Shot Image Classification

Improving Zero-Shot Generalization for CLIP with Variational Adapter

Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models

Meta-Adapter: An Online Few-shot Learner for Vision-Language Model

Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification

Ta-Adapter: Enhancing few-shot CLIP with task-aware encoders

CLIP Adaptation by Intra-modal Overlap Reduction

A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation

Rethinking Visual Content Refinement in Low-Shot CLIP Adaptation

Fine-Tuning CLIP's Last Visual Projector: A Few-Shot Cornucopia

Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models

Post-hoc Probabilistic Vision-Language Models

BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models