Enhancing Few-Shot Learning in Lightweight Models via Dual-Faceted Knowledge Distillation

Bojun Zhou,Tianyu Cheng,Jiahao Zhao,Chunkai Yan,Ling Jiang,Xinsong Zhang,Juping Gu

DOI: https://doi.org/10.3390/s24061815

IF: 3.9

2024-03-12

Sensors

Abstract:In recent computer vision research, the pursuit of improved classification performance often leads to the adoption of complex, large-scale models. However, the actual deployment of such extensive models poses significant challenges in environments constrained by limited computing power and storage capacity. Consequently, this study is dedicated to addressing these challenges by focusing on innovative methods that enhance the classification performance of lightweight models. We propose a novel method to compress the knowledge learned by a large model into a lightweight one so that the latter can also achieve good performance in few-shot classification tasks. Specifically, we propose a dual-faceted knowledge distillation strategy that combines output-based and intermediate feature-based methods. The output-based method concentrates on distilling knowledge related to base class labels, while the intermediate feature-based approach, augmented by feature error distribution calibration, tackles the potential non-Gaussian nature of feature deviations, thereby boosting the effectiveness of knowledge transfer. Experiments conducted on MiniImageNet, CIFAR-FS, and CUB datasets demonstrate the superior performance of our method over state-of-the-art lightweight models, particularly in five-way one-shot and five-way five-shot tasks.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to address the performance improvement of lightweight models in Few-Shot Classification (FSC) tasks. Specifically:

1. **Background and Challenges**:
- In current computer vision research, complex and large-scale deep learning models are typically used to improve classification performance.
- However, when deploying these large models in practical scenarios, there are numerous challenges due to limited computational resources and storage space, such as high computational costs and limited storage capacity.

2. **Research Objectives**:
- Propose a new method to compress the knowledge learned by large models and transfer it to lightweight models, enabling the latter to achieve good performance in few-shot classification tasks.
- Through this method, the goal is to narrow the performance gap between lightweight models and large models in FSC tasks.

3. **Specific Methods**:
- A Dual-Faceted Knowledge Distillation strategy is proposed, combining output-based methods and intermediate feature-based methods.
- The output-based method focuses on distilling knowledge from base category labels; the intermediate feature-based method enhances the effectiveness of knowledge transfer through feature error distribution calibration.

4. **Experimental Validation**:
- Experiments on MiniImageNet, CIFAR-FS, and CUB datasets show that this method outperforms existing lightweight models in 5-way 1-shot and 5-way 5-shot tasks.

In summary, the main contribution of this paper is the proposal of a new dual-faceted knowledge distillation method to enhance the performance of lightweight models in few-shot classification tasks.