Abstract:Recently, research efforts have been concentrated on revealing how pre-trained model makes a difference in neural network performance. Self-supervision and semi-supervised learning technologies have been extensively explored by the community and are proven to be of great potential in obtaining a powerful pre-trained model. However, these models require huge training costs (i.e., hundreds of millions of images or training iterations). In this paper, we propose to improve existing baseline networks via knowledge distillation from off-the-shelf pre-trained big powerful models. Different from existing knowledge distillation frameworks which require student model to be consistent with both soft-label generated by teacher model and hard-label annotated by humans, our solution performs distillation by only driving prediction of the student model consistent with that of the teacher model. Therefore, our distillation setting can get rid of manually labeled data and can be trained with extra unlabeled data to fully exploit capability of teacher model for better learning. We empirically find that such simple distillation settings perform extremely effective, for example, the top-1 accuracy on ImageNet-1k validation set of MobileNetV3-large and ResNet50-D can be significantly improved from 75.2% to 79% and 79.1% to 83%, respectively. We have also thoroughly analyzed what are dominant factors that affect the distillation performance and how they make a difference. Extensive downstream computer vision tasks, including transfer learning, object detection and semantic segmentation, can significantly benefit from the distilled pretrained models. All our experiments are implemented based on PaddlePaddle, codes and a series of improved pretrained models with ssld suffix are available in PaddleClas.

Self-Guidance: Improve Deep Neural Network Generalization via Knowledge Distillation

Knowledge Distillation Meets Self-Supervision

Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

Self-knowledge distillation via dropout

Data-Distortion Guided Self-Distillation for Deep Neural Networks

Self-Distillation: Towards Efficient and Compact Neural Networks

Self-Referenced Deep Learning

Progressive Network Grafting for Few-Shot Knowledge Distillation

Generalized Knowledge Distillation via Relationship Matching

Self-knowledge Distillation Based on Knowledge Transfer from Soft to Hard Examples.

Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation

Self-Distillation for Randomized Neural Networks

On Self-Distilling Graph Neural Network

Multi-target Knowledge Distillation Via Student Self-reflection

Skill-transferring Knowledge Distillation Method

Improved Knowledge Distillation via Teacher Assistant

Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones

Densely Guided Knowledge Distillation using Multiple Teacher Assistants