Abstract:Few-shot image classification aims at learning to generalize to unseen new categories from a few training samples. Transfer learning is one prominent approach to the task, which first learns a backbone from the base classes and then trains a classifier on new classes with the prior learned knowledge. Typically, the convolutional neural network (CNN) is the preferred backbone. However, when the samples are limited, the representation ability of the feature extracted by CNN will decrease, thus leading to the performance degradation of few-shot image classification. Recently, the pre-trained large-scale vision-language model like CLIP has shown non-trivial potential, which can be used as a backbone for zero or few-shot transfer on a series of downstream tasks with the prompt. To fully explore the few-shot image classification performance of vision-language models, we propose CoCoOpter, a novel "pre-training + prompt tuning + fine-tuning" paradigm based on CLIP. CoCoOpter alleviates the overfitting and ensures generalizability in unseen new categories. Specifically, it learns an input-specific neural network to relieve overfitting by drawing attention away from a specific category to each specific input sample. Then, to establish connection between the visual and textual signals, it introduces the previously learned visual representations to perform automatic prompt tuning in the middle of the pre-trained CLIP, enabling learning input-specified prompt vectors. Moreover, two learnable lightweight neural networks are added at the end of CLIP to guide information propagation between different classes by fine-tuning both the visual and textual features. We perform extensive experiments on 11 image classification datasets. The results show that CoCoOpter is more generalizable in unseen classes and achieves superior few-shot classification performance with a straightforward design.

Like Humans to Few-Shot Learning through Knowledge Permeation of Vision and Text

Knowledge-Based Fine-Grained Classification for Few-Shot Learning.

The Devil is in the Few Shots: Iterative Visual Knowledge Completion for Few-shot Learning

Knowledge-Guided Semantic Transfer Network for Few-Shot Image Recognition

Multi-directional Knowledge Transfer for Few-Shot Learning

Binocular Mutual Learning for Improving Few-shot Classification

Multi-domain few-shot image recognition with knowledge transfer

Knowledge Graph Enhanced Multimodal Learning for Few-shot Visual Recognition

BDLA: Bi-directional local alignment for few-shot learning

Adapting Vision-Language Models via Learning to Inject Knowledge

Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models

CoCoOpter: Pre-train, prompt, and fine-tune the vision-language model for few-shot image classification

Mining Open Semantics from CLIP: A Relation Transition Perspective for Few-Shot Learning

Few-Shot Scene Classification of Optical Remote Sensing Images Leveraging Calibrated Pretext Tasks

Decoupling Representation and Knowledge for Few-Shot Intent Classification and Slot Filling

LPN: Language-guided Prototypical Network for few-shot classification

Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognition

Few-shot Learning for Multi-Modality Tasks

Gestalt-Guided Image Understanding for Few-Shot Learning

Learning from One and Only One Shot

Less is More: A Closer Look at Semantic-based Few-Shot Learning