Abstract:Knowledge distillation, which involves extracting the "dark knowledge" from a teacher network to guide the learning of a student network, has emerged as an important technique for model compression and transfer learning. Unlike previous works that exploit architecture-specific cues such as activation and attention for distillation, here we wish to explore a more general and model-agnostic approach for extracting "richer dark knowledge" from the pre-trained teacher model. We show that the seemingly different self-supervision task can serve as a simple yet powerful solution. For example, when performing contrastive learning between transformed entities, the noisy predictions of the teacher network reflect its intrinsic composition of semantic and pose information. By exploiting the similarity between those self-supervision signals as an auxiliary task, one can effectively transfer the hidden information from the teacher to the student. In this paper, we discuss practical ways to exploit those noisy self-supervision signals with selective transfer for distillation. We further show that self-supervision signals improve conventional distillation with substantial gains under few-shot and noisy-label scenarios. Given the richer knowledge mined from self-supervision, our knowledge distillation approach achieves state-of-the-art performance on standard benchmarks, i.e., CIFAR100 and ImageNet, under both similar-architecture and cross-architecture settings. The advantage is even more pronounced under the cross-architecture setting, where our method outperforms the state of the art CRD by an average of 2.3% in accuracy rate on CIFAR100 across six different teacher-student pairs.

Self-Attention Guided Advice Distillation in Multi-Agent Deep Reinforcement Learning

Agent-Aware Training for Agent-Agnostic Action Advising in Deep Reinforcement Learning

Action Advising with Advice Imitation in Deep Reinforcement Learning

A federated advisory teacher–student framework with simultaneous learning agents

Multi-Agent Advisor Q-Learning

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

Multi-Advisor Reinforcement Learning

Shaping Advice in Deep Multi-Agent Reinforcement Learning

A GNN-based teacher–student framework with multi-advice

A location-based advising method in teacher–student frameworks

Neural-to-Tree Policy Distillation with Policy Improvement Criterion

Learning from Multiple Independent Advisors in Multi-agent Reinforcement Learning

Learning to Advise and Learning from Advice in Cooperative Multiagent Reinforcement Learning.

Tree-like Decision Distillation

Dual Policy Distillation

A Q-values Sharing Framework for Multiagent Reinforcement Learning under Budget Constraint

Ask-AC: An Initiative Advisor-in-the-Loop Actor-Critic Framework

Methodical Advice Collection and Reuse in Deep Reinforcement Learning

Knowledge Distillation Meets Self-Supervision

Learning Multi-Agent Communication with Double Attentional Deep Reinforcement Learning

Dual teachers for self-knowledge distillation