Abstract:Knowledge distillation guides student networks’ training and enhances their performance through excellent teacher networks. However, along with the performance advantages, knowledge distillation also entails a huge computational burden, sometimes tens or even hundreds of times that of traditional training methods. So, this paper designs a book-based knowledge distillation (BookKD) to minimize the costs of knowledge distillation while improving performance. First, a decoupling-based knowledge distillation framework is designed. By decoupling the traditional knowledge distillation process into two independent sub-processes, book-making and book-learning, knowledge distillation can be completed with little resource consumption. Second, a book-making method based on knowledge ensemble and knowledge regularization is developed, which makes books by organizing and processing the knowledge generated by teachers. These books can replace these teachers to provide sufficient knowledge with little distillation costs. Finally, a book-learning method based on entropy dynamic adjustment and label smoothing is designed. The entropy dynamic adjustment optimizes the training loss and mitigates student networks’ difficulty in learning books. Label smoothing alleviates the student network’s over-confidence in ground truth labels, which increases its attention to the class similarity knowledge in books. BookKD is tested on three image classification datasets, CIFAR100, ImageNet and ImageNet100, and an object detection dataset PASCAL VOC 2007. The experiment results indicate the advantages of BookKD in reducing distillation costs and improving distillation performance.

Knowledge Distillation for Efficient Sequences of Training Runs

Practical Insights into Knowledge Distillation for Pre-Trained Models

Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability

Knowledge Distillation Performs Partial Variance Reduction

Comparative Knowledge Distillation

QEKD: Query-Efficient and Data-Free Knowledge Distillation from Black-box Models.

An Embarrassingly Simple Approach for Knowledge Distillation

Stage-by-stage Knowledge Distillation

Residual Error Based Knowledge Distillation

Knowledge Condensation Distillation

Lightweight Self-Knowledge Distillation with Multi-source Information Fusion

Reinforced Knowledge Distillation for Time Series Regression

Dynamic Knowledge Distillation for Pre-trained Language Models

Data Efficient Stagewise Knowledge Distillation

Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation

Knowledge Distillation Via Channel Correlation Structure

BookKD: A Novel Knowledge Distillation for Reducing Distillation Costs by Decoupling Knowledge Generation and Learning

Densely Distilling Cumulative Knowledge for Continual Learning

Learning from a Lightweight Teacher for Efficient Knowledge Distillation

A Closer Look at Knowledge Distillation with Features, Logits, and Gradients

Elastic Knowledge Distillation by Learning from Recollection.