Abstract:As a specific form of knowledge distillation (KD), self-knowledge distillation enables a student network to progressively distill its own knowledge without relying on a pretrained, complex teacher network; however, recent studies of self-KD have discovered that additional dark knowledge captured by auxiliary architecture or data augmentation could create better soft targets for enhancing the network but at the cost of significantly more computations and/or parameters. Moreover, most existing self-KD methods extract the soft label as a supervisory signal from individual input samples, which overlooks the knowledge of relationships among categories. Inspired by human associative learning, we propose a simple yet effective self-KD method named associative learning for self-distillation (ALSD), which progressively distills richer knowledge regarding the relationships between categories across independent samples. Specifically, in the process of distillation, the propagation of knowledge is weighted based on the intersample relationship between associated samples generated in different minibatches, which are progressively estimated with the current network. In this way, our ALSD framework achieves knowledge ensembling progressively across multiple samples using a single network, resulting in minimal computational and memory overhead compared to existing ensembling methods. Extensive experiments demonstrate that our ALSD method consistently boosts the classification performance of various architectures on multiple datasets. Notably, ALSD pushes forward the self-KD performance to 80.10% on CIFAR-100, which exceeds the standard backpropagation by 4.81%. Furthermore, we observe that the proposed method shows comparable performance with the state-of-the-art knowledge distillation methods without the pretrained teacher network.

ADCL: Adversarial Distilled Contrastive Learning on lightweight models for self-supervised image classification

DCCD: Reducing Neural Network Redundancy Via Distillation

Tolerant Self-Distillation for Image Classification

Classifier Reuse-Based Contrastive Knowledge Distillation

Category contrastive distillation with self-supervised classification

Data-Free Adversarial Distillation

Knowledge Distillation Meets Self-Supervision

Learning Efficient Detector with Semi-supervised Adaptive Distillation

Lightweight High-Resolution Remote Sensing Scene Classification via Adaptive Enhanced Knowledge Distillation.

Pixel-Wise Contrastive Distillation

Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation

Image Classification Based on Self-Distillation

Distilling Visual Priors from Self-Supervised Learning

Self-Knowledge Distillation via Progressive Associative Learning

DenseCL: A Simple Framework for Self-Supervised Dense Visual Pre-Training

Online Knowledge Distillation Via Mutual Contrastive Learning for Visual Recognition

Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones

Image Quality Assessment Based on Self-Supervised Learning and Knowledge Distillation

A Novel Self-Learning Network Integrating Contrastive Learning, Perceptual Learning and Masked Image Modelling

Learning Lightweight Lane Detection CNNs by Self Attention Distillation

LTD: Low Temperature Distillation for Robust Adversarial Training