Abstract:Advancements in open-source pre-trained backbones make it relatively easy to fine-tune a model for new tasks. However, this lowered entry barrier poses potential risks, e.g., bad actors developing models for harmful applications. A question arises: Is possible to develop a pre-trained model that is difficult to fine-tune for certain downstream tasks? To begin studying this, we focus on few-shot classification (FSC). Specifically, we investigate methods to make FSC more challenging for a set of restricted classes while maintaining the performance of other classes. We propose to meta-learn over the pre-trained backbone in a manner that renders it a ''poor initialization''. Our proposed Learning to Obstruct (LTO) algorithm successfully obstructs four FSC methods across three datasets, including ImageNet and CIFAR100 for image classification, as well as CelebA for attribute classification.

What problem does this paper attempt to address?

### The problems the paper attempts to solve This paper aims to explore whether it is possible to develop a pre - trained model that is difficult to fine - tune for certain specific downstream tasks, so as to deal with potential security risks in the context of the increasing popularity of open - source code and pre - trained models. Specifically, the author focuses on the Few - Shot Classification (FSC) problem and proposes a method to make the FSC performance of the pre - trained model worse on specific restricted classes while keeping the performance unchanged on other classes. ### Research background and motivation With the development of open - source pre - trained models, great progress has been made in the fields of computer vision and machine learning. These models have improved the reproducibility of research and fair comparison, but they have also brought new challenges. For example, the ability to quickly fine - tune models to adapt to new tasks has lowered the threshold for developing computer vision systems, which may be exploited by malicious users to develop harmful applications, such as face recognition systems that violate privacy. ### Specific problem description To meet this challenge, the author poses the following questions: - Is it possible to develop a pre - trained model that is difficult to fine - tune for certain specific downstream tasks? If this can be achieved successfully, the pre - trained model can support scientific research while solving security problems. For this purpose, the author chooses to start from the Few - Shot Classification (FSC) task and study how to make the pre - trained model perform poorly on a set of restricted classes while remaining competitive on other classes. ### Method overview To solve the above problems, the author proposes the Learning to Obstruct (LTO) algorithm, which is a meta - learning algorithm. By modifying the pre - trained model parameters to make them "badly initialized", it hinders the learning of FSC methods on restricted classes. The core idea of the LTO algorithm is to optimize an objective function so that the performance of the model on restricted classes decreases while the performance on non - restricted classes remains unchanged. ### Experimental verification The author verifies the effectiveness of LTO through multiple experiments, including: 1. Experiments on classical few - shot classification methods (such as ProtoNet, MetaOptNet). 2. Experiments on CLIP - based few - shot classification methods (such as CoOp, TipAdapter). The experimental results show that LTO can significantly reduce the accuracy of the model on restricted classes without affecting the performance of other classes, thus effectively hindering the learning of specific tasks. ### Summary The main contributions of this paper are: - Proposing a task setting for hindering the learning of specific classes. - Designing and implementing the LTO algorithm, which adjusts the pre - trained model parameters through meta - learning to achieve the hindrance of specific classes. - Verifying the effectiveness of LTO through extensive experiments and demonstrating its application potential on multiple FSC algorithms and datasets. Through this method, the author provides a new idea for the safe release of pre - trained models, which both supports scientific research and avoids potential risks.

Learning to Obstruct Few-Shot Image Classification over Restricted Classes

Interventional Few-Shot Learning

Generalizable and Discriminative Representations for Adversarially Robust Few-Shot Learning

Fine-Grained Few Shot Learning with Foreground Object Transformation

TAFSSL: Task-Adaptive Feature Sub-Space Learning for few-shot classification

What Makes for Effective Few-shot Point Cloud Classification?

Learning to focus: cascaded feature matching network for few-shot image recognition

Federated Few-Shot Learning with Adversarial Learning

Learning What Not to Segment: A New Perspective on Few-Shot Segmentation

Learning Relative Feature Displacement for Few-Shot Open-Set Recognition

Few-Shot Fine-Grained Image Classification via Multi-Frequency Neighborhood and Double-Cross Modulation

Transductive Few-Shot Classification on the Oblique Manifold

Low-Mid Adversarial Perturbation against Unauthorized Face Recognition System

CLOSER: Towards Better Representation Learning for Few-Shot Class-Incremental Learning

Coarse-to-fine pseudo supervision guided meta-task optimization for few-shot object classification

Adaptive Local Feature Matching for Few-shot Fine-grained Image Recognition

Controllable Relation Disentanglement for Few-Shot Class-Incremental Learning

CORE: CORrelation-Guided Feature Enhancement for Few-Shot Image Classification

Few-Shot Object Detection in Unseen Domains

FCert: Certifiably Robust Few-Shot Classification in the Era of Foundation Models

Embedding enhancement with foreground feature alignment and primitive knowledge for few-shot learning