Abstract:As a specific form of knowledge distillation (KD), self-knowledge distillation enables a student network to progressively distill its own knowledge without relying on a pretrained, complex teacher network; however, recent studies of self-KD have discovered that additional dark knowledge captured by auxiliary architecture or data augmentation could create better soft targets for enhancing the network but at the cost of significantly more computations and/or parameters. Moreover, most existing self-KD methods extract the soft label as a supervisory signal from individual input samples, which overlooks the knowledge of relationships among categories. Inspired by human associative learning, we propose a simple yet effective self-KD method named associative learning for self-distillation (ALSD), which progressively distills richer knowledge regarding the relationships between categories across independent samples. Specifically, in the process of distillation, the propagation of knowledge is weighted based on the intersample relationship between associated samples generated in different minibatches, which are progressively estimated with the current network. In this way, our ALSD framework achieves knowledge ensembling progressively across multiple samples using a single network, resulting in minimal computational and memory overhead compared to existing ensembling methods. Extensive experiments demonstrate that our ALSD method consistently boosts the classification performance of various architectures on multiple datasets. Notably, ALSD pushes forward the self-KD performance to 80.10% on CIFAR-100, which exceeds the standard backpropagation by 4.81%. Furthermore, we observe that the proposed method shows comparable performance with the state-of-the-art knowledge distillation methods without the pretrained teacher network.

Neighbor Self-Knowledge Distillation

DCCD: Reducing Neural Network Redundancy Via Distillation

Self-Knowledge Distillation via Progressive Associative Learning

Lightweight Self-Knowledge Distillation with Multi-source Information Fusion

Self-Referenced Deep Learning

Self-Distillation: Towards Efficient and Compact Neural Networks

Knowledge Distillation Meets Self-Supervision

An Embarrassingly Simple Approach for Knowledge Distillation

From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels

Memory-Replay Knowledge Distillation

Self-knowledge distillation via dropout

Self-Distillation for Randomized Neural Networks

Dual teachers for self-knowledge distillation

Collaborative Knowledge Distillation

Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones

Deep Collective Knowledge Distillation

On Self-Distilling Graph Neural Network

Online Knowledge Distillation via Collaborative Learning

Multi-target Knowledge Distillation Via Student Self-reflection

Adaptive Cross-Architecture Mutual Knowledge Distillation