Knowledge Distillation Meets Self-Supervision

Guodong Xu,Ziwei Liu,Xiaoxiao Li,Chen Change Loy

DOI: https://doi.org/10.48550/arXiv.2006.07114

2020-07-13

Abstract:Knowledge distillation, which involves extracting the "dark knowledge" from a teacher network to guide the learning of a student network, has emerged as an important technique for model compression and transfer learning. Unlike previous works that exploit architecture-specific cues such as activation and attention for distillation, here we wish to explore a more general and model-agnostic approach for extracting "richer dark knowledge" from the pre-trained teacher model. We show that the seemingly different self-supervision task can serve as a simple yet powerful solution. For example, when performing contrastive learning between transformed entities, the noisy predictions of the teacher network reflect its intrinsic composition of semantic and pose information. By exploiting the similarity between those self-supervision signals as an auxiliary task, one can effectively transfer the hidden information from the teacher to the student. In this paper, we discuss practical ways to exploit those noisy self-supervision signals with selective transfer for distillation. We further show that self-supervision signals improve conventional distillation with substantial gains under few-shot and noisy-label scenarios. Given the richer knowledge mined from self-supervision, our knowledge distillation approach achieves state-of-the-art performance on standard benchmarks, i.e., CIFAR100 and ImageNet, under both similar-architecture and cross-architecture settings. The advantage is even more pronounced under the cross-architecture setting, where our method outperforms the state of the art CRD by an average of 2.3% in accuracy rate on CIFAR100 across six different teacher-student pairs.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to extract more abundant "dark knowledge" from pre - trained teacher models to guide the learning of student models. Specifically, the paper proposes a new framework SSKD (Self - Supervised Knowledge Distillation), which enhances the effect of traditional knowledge distillation methods by using self - supervised learning as an auxiliary task. Unlike traditional knowledge distillation methods that mainly focus on model - architecture - specific cues (such as activations and attention), SSKD aims to explore a more general and model - independent method, so as to be able to extract more abundant implicit information from teacher models.

### Core problems of the paper
1. **How to extract more abundant dark knowledge from teacher models**: Existing knowledge distillation methods usually rely on the intermediate representations of teacher models, which are often task - specific and model - architecture - specific. SSKD enables student models to learn more structured knowledge from teacher models by introducing self - supervised learning tasks, rather than just knowledge for classification tasks.

2. **How to conduct effective knowledge transfer between different model architectures**: Traditional knowledge distillation methods often perform poorly between different architectures because the knowledge they transfer is highly dependent on specific model architectures. SSKD allows student models to search for intermediate representations that are most suitable for their own architectures in a more flexible solution space by only transmitting the output of the last layer.

3. **How to improve the generalization ability of student models in few - shot and noisy - label scenarios**: The paper points out that by using self - supervised learning tasks, SSKD not only achieves state - of - the - art performance on standard benchmark datasets, but also shows significant advantages in few - shot and noisy - label scenarios.

### Solutions
- **Self - supervised learning as an auxiliary task**: SSKD generates self - supervised signals by adding a lightweight auxiliary module in the teacher model and using self - supervised learning tasks (such as contrastive learning). These signals contain rich structured knowledge and can be used as auxiliary tasks to guide the learning of student models.

- **Selective transmission**: In order to deal with the noise in self - supervised predictions, SSKD adopts a selective transmission strategy, that is, only transmitting the correct predictions and a top - ranked part of the wrong predictions, thereby avoiding the negative impact of extremely wrong predictions on the learning process.

- **Multi - task learning**: SSKD requires student models not only to imitate the classification output of teacher models, but also to imitate the self - supervised output of teacher models. This multi - task learning method enables student models to learn more comprehensive knowledge from multiple perspectives.

### Experimental results
- **Performance on standard benchmark datasets**: SSKD achieves state - of - the - art performance on both the CIFAR100 and ImageNet datasets, especially in the knowledge transfer tasks between different architectures, with an average accuracy rate 2.3% higher than existing methods.

- **Advantages in few - shot and noisy - label scenarios**: SSKD shows stronger generalization ability in few - shot and noisy - label scenarios, which indicates that self - supervised learning tasks do help student models learn more abundant knowledge.

In conclusion, by introducing self - supervised learning as an auxiliary task, this paper successfully solves the problems of how to extract more abundant dark knowledge from teacher models and conduct effective knowledge transfer between different model architectures.