Abstract:Graph, such as citation networks, social networks, and transportation networks, are prevalent in the real world. Graph Neural Networks (GNNs) have gained widespread attention for their robust expressiveness and exceptional performance in various graph applications. However, the efficacy of GNNs is heavily reliant on sufficient data labels and complex network models, with the former obtaining hardly and the latter computing costly. To address the labeled data scarcity and high complexity of GNNs, Knowledge Distillation (KD) has been introduced to enhance existing GNNs. This technique involves transferring the soft-label supervision of the large teacher model to the small student model while maintaining prediction performance. This survey offers a comprehensive overview of Graph-based Knowledge Distillation methods, systematically categorizing and summarizing them while discussing their limitations and future directions. This paper first introduces the background of graph and KD. It then provides a comprehensive summary of three types of Graph-based Knowledge Distillation methods, namely Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for GNNs (GKD), and Self-Knowledge Distillation based Graph-based Knowledge Distillation (SKD). Each type is further divided into knowledge distillation methods based on the output layer, middle layer, and constructed graph. Subsequently, various algorithms' ideas are analyzed and compared, concluding with the advantages and disadvantages of each algorithm supported by experimental results. In addition, the applications of graph-based knowledge distillation in CV, NLP, RS, and other fields are listed. Finally, the graph-based knowledge distillation is summarized and prospectively discussed. We have also released related resources at https://github.com/liujing1023/Graph-based-Knowledge-Distillation.

Enhanced Scalable Graph Neural Network via Knowledge Distillation

Knowledge Distillation Improves Graph Structure Augmentation for Graph Neural Networks

GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent Inference

Boosting Graph Neural Networks via Adaptive Knowledge Distillation

Multi-Scale Distillation from Multiple Graph Neural Networks

Shared Growth of Graph Neural Networks via Free-direction Knowledge Distillation

Knowledge Distillation Via Adaptive Meta-Learning for Graph Neural Network

Knowledge Distillation on Graphs: A Survey

Frameless Graph Knowledge Distillation

Shared Growth of Graph Neural Networks via Prompted Free-direction Knowledge Distillation

Graph-based Knowledge Distillation: A survey and experimental evaluation

Fine-Grained Learning Behavior-Oriented Knowledge Distillation for Graph Neural Networks

FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks

Online Adversarial Knowledge Distillation for Graph Neural Networks

On Representation Knowledge Distillation for Graph Neural Networks

Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation

Decoupled graph knowledge distillation: A general logits-based method for learning MLPs on graphs

Online Cross-Layer Knowledge Distillation on Graph Neural Networks with Deep Supervision

Learning to Distill Graph Neural Networks.

Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework

Scalable Graph Neural Networks Via Bidirectional Propagation.