Abstract:Few-shot learning poses a critical challenge due to the deviation problem caused by the scarcity of available samples. In this work, we aim to address deviations in both feature representations and prototypes. To achieve this, we propose a cross-modal de-deviation framework that leverages class semantic information to provide robust prior knowledge for the samples. This framework begins with a visual-to-semantic autoencoder trained on the labeled samples to predict semantic features for the unlabeled samples. Then, we devise a binary linear programming model to match the initial prototypes with the cluster centers of the unlabeled samples. To circumvent potential mismatches between the cluster centers and the initial prototypes, we perform the label assignment process in the semantic space by transforming the cluster centers into semantic representations and utilizing the class ground truth semantic features as reference points. Moreover, we model a linear classifier with the concatenation of the refined prototypes and the class ground truth semantic features serving as the initial weights. Then we propose a novel optimization strategy based on the alternating least squares (ALS) model. From the ALS model, we can derive two closed-form solutions regarding to the features and weights, facilitating alternative optimization of them. Extensive experiments conducted on few-shot learning benchmarks demonstrate the competitive advantages of our CMDD method over the state-of-the-art approaches, confirming its effectiveness in reducing deviation. The code is available at: https://github.com/pmhDL/CMDD.git.

Multimodal variational contrastive learning for few-shot classification

Multimodal few-shot classification without attribute embedding

Cross-Modal De-Deviation for Enhancing Few-Shot Classification

Contrastive Knowledge-Augmented Self-Distillation Approach for Few-Shot Learning

Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models

Adaptive Cross-Modal Few-Shot Learning

Momentum Memory Contrastive Learning for Transfer-Based Few-Shot Classification

Supervised Momentum Contrastive Learning for Few-Shot Classification

Enhancing Few-Shot Classification without Forgetting through Multi-Level Contrastive Constraints

Transductive Semantic Decoupling Double Variational Inference for Few-Shot Classification

Auto-view Contrastive Learning for Few-Shot Image Recognition

Bimodal semantic fusion prototypical network for few-shot classification

Boosting Few-Shot Classification with View-Learnable Contrastive Learning

Better Integrating Vision and Semantics for Improving Few-shot Classification

Dual-stream Multi-Modal Graph Neural Network for Few-Shot Learning

MHA-WoML: Multi-head Attention and Wasserstein-OT for Few-Shot Learning

Cross-Modal Mapping: Eliminating the Modality Gap for Few-Shot Image Classification

Ensembling Multi-View Discriminative Semantic Feature for Few-Shot Classification

Multimodal Prototypical Networks for Few-shot Learning

Multiscale Object Contrastive Learning-Derived Few-Shot Object Detection in VHR Imagery

Semantic-Based Few-Shot Learning by Interactive Psychometric Testing