Abstract:As a specific form of knowledge distillation (KD), self-knowledge distillation enables a student network to progressively distill its own knowledge without relying on a pretrained, complex teacher network; however, recent studies of self-KD have discovered that additional dark knowledge captured by auxiliary architecture or data augmentation could create better soft targets for enhancing the network but at the cost of significantly more computations and/or parameters. Moreover, most existing self-KD methods extract the soft label as a supervisory signal from individual input samples, which overlooks the knowledge of relationships among categories. Inspired by human associative learning, we propose a simple yet effective self-KD method named associative learning for self-distillation (ALSD), which progressively distills richer knowledge regarding the relationships between categories across independent samples. Specifically, in the process of distillation, the propagation of knowledge is weighted based on the intersample relationship between associated samples generated in different minibatches, which are progressively estimated with the current network. In this way, our ALSD framework achieves knowledge ensembling progressively across multiple samples using a single network, resulting in minimal computational and memory overhead compared to existing ensembling methods. Extensive experiments demonstrate that our ALSD method consistently boosts the classification performance of various architectures on multiple datasets. Notably, ALSD pushes forward the self-KD performance to 80.10% on CIFAR-100, which exceeds the standard backpropagation by 4.81%. Furthermore, we observe that the proposed method shows comparable performance with the state-of-the-art knowledge distillation methods without the pretrained teacher network.

Self-knowledge Distillation Based on Knowledge Transfer from Soft to Hard Examples.

Soft Hybrid Knowledge Distillation Against Deep Neural Networks

Self-Knowledge Distillation via Progressive Associative Learning

Knowledge Distillation Meets Self-Supervision

What Knowledge Gets Distilled in Knowledge Distillation?

Multi-instance semantic similarity transferring for knowledge distillation

Teacher-student collaborative knowledge distillation for image classification

Skill-transferring Knowledge Distillation Method

Learning from Teacher’s Failure: A Reflective Learning Paradigm for Knowledge Distillation

Self-Knowledge Distillation in Natural Language Processing

Collaborative Knowledge Distillation

Multi-target Knowledge Distillation Via Student Self-reflection

Similarity Transfer for Knowledge Distillation

Student-Oriented Teacher Knowledge Refinement for Knowledge Distillation

Exploring the Knowledge Transferred by Response-Based Teacher-Student Distillation

An Embarrassingly Simple Approach for Knowledge Distillation

Learning Student-Friendly Teacher Networks for Knowledge Distillation

Student-friendly Knowledge Distillation

Neighbor Self-Knowledge Distillation

Self-Knowledge Distillation with Learning from Role-Model Samples.

Two-Stage Approach for Targeted Knowledge Transfer in Self-Knowledge Distillation