Abstract:Learning generalized representations from limited training samples is crucial for applying deep neural networks in low-resource scenarios. Recently, methods based on contrastive language-image pretraining (CLIP) have exhibited promising performance in few-shot adaptation tasks. To avoid catastrophic forgetting and overfitting caused by few-shot fine-tuning, existing works usually freeze the parameters of CLIP pretrained on large-scale datasets, overlooking the possibility that some parameters might not be suitable for downstream tasks. To this end, we revisit CLIP's visual encoder with a specific focus on its distinctive attention pooling layer, which performs a spatial weighted-sum of the dense feature maps. Given that dense feature maps contain meaningful semantic information, and different semantics hold varying importance for diverse downstream tasks (such as prioritizing semantics like ears and eyes in pet classification tasks rather than side mirrors), using the same weighted-sum operation for dense features across different few-shot tasks might not be appropriate. Hence, we propose fine-tuning the parameters of the attention pooling layer during the training process to encourage the model to focus on task-specific semantics. In the inference process, we perform residual blending between the features pooled by the fine-tuned and the original attention pooling layers to incorporate both the few-shot knowledge and the pretrained CLIP's prior knowledge. We term this method as semantic-aware fine-tuning (). is effective in enhancing the conventional few-shot CLIP and is compatible with the existing adapter approach (termed ). Extensive experiments on 11 benchmarks demonstrate that both and significantly outperform the second-best method by + 1.51 % and + 2.38 % in the one-shot setting and by + 0.48 % and + 1.37 % in the four-shot setting, respectively.

Joint contrastive learning for prompt-based few-shot language learners

ConsPrompt: Exploiting Contrastive Samples for Fewshot Prompt Learning

A Soft Contrastive Learning-based Prompt Model for Few-shot Sentiment Analysis

Deeply Coupled Cross-Modal Prompt Learning

Unified Prompt Learning Makes Pre-Trained Language Models Better Few-Shot Learners

SPContrastNet: A Self-Paced Contrastive Learning Model for Few-Shot Text Classification

A Unified Label-Aware Contrastive Learning Framework for Few-Shot Named Entity Recognition

ContrastNet: A Contrastive Learning Framework for Few-Shot Text Classification

Ontology-enhanced Prompt-tuning for Few-shot Learning

Hierarchy-Aware Interactive Prompt Learning for Few-Shot Classification

FewJoint: few-shot learning for joint dialogue understanding

Adaptive Prompt Learning-Based Few-Shot Sentiment Analysis

Less is More: A Closer Look at Semantic-based Few-Shot Learning

Enhancing Few-Shot CLIP With Semantic-Aware Fine-Tuning

DiffCLIP: Few-shot Language-driven Multimodal Classifier

CLG: Contrastive Label Generation with Knowledge for Few-Shot Learning

Enhancing Few-Shot Classification without Forgetting through Multi-Level Contrastive Constraints

Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners

Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene

Unified Contrastive Learning in Image-Text-Label Space

A Two-Stage Prediction-Aware Contrastive Learning Framework for Multi-Intent NLU