Abstract:As a specific form of knowledge distillation (KD), self-knowledge distillation enables a student network to progressively distill its own knowledge without relying on a pretrained, complex teacher network; however, recent studies of self-KD have discovered that additional dark knowledge captured by auxiliary architecture or data augmentation could create better soft targets for enhancing the network but at the cost of significantly more computations and/or parameters. Moreover, most existing self-KD methods extract the soft label as a supervisory signal from individual input samples, which overlooks the knowledge of relationships among categories. Inspired by human associative learning, we propose a simple yet effective self-KD method named associative learning for self-distillation (ALSD), which progressively distills richer knowledge regarding the relationships between categories across independent samples. Specifically, in the process of distillation, the propagation of knowledge is weighted based on the intersample relationship between associated samples generated in different minibatches, which are progressively estimated with the current network. In this way, our ALSD framework achieves knowledge ensembling progressively across multiple samples using a single network, resulting in minimal computational and memory overhead compared to existing ensembling methods. Extensive experiments demonstrate that our ALSD method consistently boosts the classification performance of various architectures on multiple datasets. Notably, ALSD pushes forward the self-KD performance to 80.10% on CIFAR-100, which exceeds the standard backpropagation by 4.81%. Furthermore, we observe that the proposed method shows comparable performance with the state-of-the-art knowledge distillation methods without the pretrained teacher network.

Data-Distortion Guided Self-Distillation for Deep Neural Networks

Data-Distortion Guided Self-Distillation for Deep Neural Networks

DCCD: Reducing Neural Network Redundancy Via Distillation

Self-Distillation: Towards Efficient and Compact Neural Networks

Progressive Network Grafting for Few-Shot Knowledge Distillation

Self-Distillation for Randomized Neural Networks

Self-knowledge distillation via dropout

Knowledge Fusion Distillation: Improving Distillation with Multi-scale Attention Mechanisms

Tolerant Self-Distillation for Image Classification

Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

Channel Distillation: Channel-Wise Attention for Knowledge Distillation

Self-Guidance: Improve Deep Neural Network Generalization via Knowledge Distillation

On Self-Distilling Graph Neural Network

Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones

Knowledge Distillation Meets Self-Supervision

Self-Referenced Deep Learning

Channel-wise Knowledge Distillation for Dense Prediction

Self-Knowledge Distillation via Progressive Associative Learning

Rethinking Data Distillation: Do Not Overlook Calibration