Abstract:In the past few years, large-scale pre-trained vision-language models like CLIP have achieved tremendous success in various fields. Naturally, how to transfer the rich knowledge in such huge pre-trained models to downstream tasks and datasets becomes a hot topic. During downstream adaptation, the most challenging problems are overfitting and catastrophic forgetting, which can cause the model to overly focus on the current data and lose more crucial domain-general knowledge. Existing works use classic regularization techniques to solve the problems. As solutions become increasingly complex, the ever-growing storage and inference costs are also a significant problem that urgently needs to be addressed. While in this paper, we start from an observation that proper random noise can suppress overfitting and catastrophic forgetting. Then we regard quantization error as a kind of noise, and explore quantization for regularizing vision-language model, which is quite efficiency and effective. Furthermore, to improve the model's generalization capability while maintaining its specialization capacity at minimal cost, we deeply analyze the characteristics of the weight distribution in prompts, conclude several principles for quantization module design and follow such principles to create several competitive baselines. The proposed method is significantly efficient due to its inherent lightweight nature, making it possible to adapt on extremely resource-limited devices. Our method can be fruitfully integrated into many existing approaches like MaPLe, enhancing accuracy while reducing storage overhead, making it more powerful yet versatile. Extensive experiments on 11 datasets shows great superiority of our method sufficiently. Code is available at <a class="link-external link-https" href="https://github.com/beyondhtx/QPrompt" rel="external noopener nofollow">this https URL</a>.

You Never Know: Quantization Induces Inconsistent Biases in Vision-Language Foundation Models

Compressed Models Decompress Race Biases: What Quantized Models Forget for Fair Face Recognition

When Quantization Affects Confidence of Large Language Models?

Contrastive Quant: Quantization Makes Stronger Contrastive Learning

Quantized Prompt for Efficient Generalization of Vision-Language Models

A Multidimensional Analysis of Social Biases in Vision Transformers

Contrastive Quant

Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models

Quantization Effects on Neural Networks Perception: How would quantization change the perceptual field of vision models?

Foundations of Large Language Model Compression -- Part 1: Weight Quantization

Seeing through bag-of-visual-word glasses: towards understanding quantization effects in feature extraction methods

Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models

Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation

Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective

Survey of Social Bias in Vision-Language Models

QuIP: 2-Bit Quantization of Large Language Models With Guarantees

How Does Quantization Affect Multilingual LLMs?

Saliency Assisted Quantization for Neural Networks

NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers

Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners

Mixed Non-linear Quantization for Vision Transformers