Abstract:Graph distillation has emerged as a promising avenue to enable scalable training of GNNs by compressing the training dataset while preserving essential graph characteristics. Our study uncovers significant shortcomings in current graph distillation techniques. First, the majority of the algorithms paradoxically require training on the full dataset to perform distillation. Second, due to their gradient-emulating approach, these methods require fresh distillation for any change in hyperparameters or GNN architecture, limiting their flexibility and reusability. Finally, they fail to achieve substantial size reduction due to synthesizing fully-connected, edge-weighted graphs. To address these challenges, we present Bonsai, a novel graph distillation method empowered by the observation that \textit{computation trees} form the fundamental processing units of message-passing GNNs. Bonsai distills datasets by encoding a careful selection of \textit{exemplar} trees that maximize the representation of all computation trees in the training set. This unique approach imparts Bonsai as the first linear-time, model-agnostic graph distillation algorithm for node classification that outperforms existing baselines across $6$ real-world datasets on accuracy, while being $22$ times faster on average. Bonsai is grounded in rigorous mathematical guarantees on the adopted approximation strategies making it robust to GNN architectures, datasets, and parameters.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are several key limitations of existing graph distillation techniques in node classification tasks. Specifically, these problems include: 1. **Requiring training on the complete dataset**: Most existing graph distillation algorithms need to train the target GNN (Graph Neural Network) on the complete training dataset first before distillation can be carried out. This design violates the basic premise of graph distillation, which is to reduce the consumption of computational resources and time. 2. **Generating fully - connected graphs**: Existing graph distillation methods tend to generate fully - connected, weighted graphs, which results in no significant reduction in the number of edges in the distilled graph, and sometimes even more edges than the original graph (see Table 2). This is inconsistent with the computational structure of GNNs, because the forward - propagation computational complexity of GNNs mainly depends on the number of edges. 3. **Model - specific distillation**: Due to the methods relying on gradient simulation, existing graph distillation algorithms are very sensitive to GNN architectures and hyperparameters. Any change in architecture or hyperparameters requires re - distillation, which limits their flexibility and reusability. To solve the above problems, the paper proposes Bonsai, a novel graph distillation method. The main contributions of Bonsai are as follows: - **Gradient - free distillation**: Bonsai achieves a distillation process independent of hyperparameters and model architectures by imitating the input data distribution processed by message - passing GNNs, rather than replicating the gradient trajectory. - **Novel algorithm design**: Bonsai is based on the observation that message - passing GNNs decompose graphs into root - calculation trees and select a set of representative calculation trees (called exemplar trees), which are located in dense areas and can represent the entire dataset. - **Empirical evaluation**: Through rigorous benchmarking on six real - world datasets, Bonsai shows higher prediction accuracy, faster distillation speed (on average 22 times faster), and stronger robustness to different GNN architectures and datasets. In conclusion, Bonsai aims to overcome the limitations of existing graph distillation methods and provide a more efficient, more flexible, and more representative graph distillation solution.

Bonsai: Gradient-free Graph Distillation for Node Classification

Mirage: Model-Agnostic Graph Distillation for Graph Classification

Classifying Nodes in Graphs without GNNs

Fair Graph Distillation

Graph Distillation with Eigenbasis Matching

GraphDIVE: Graph Classification by Mixture of Diverse Experts

Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs

Compressing Deep Graph Neural Networks via Adversarial Knowledge Distillation

Online Adversarial Knowledge Distillation for Graph Neural Networks

Knowledge Distillation on Graphs: A Survey

Enhanced Scalable Graph Neural Network via Knowledge Distillation

Graph neural networks meet with distributed graph partitioners and reconciliations

Rethinking and Accelerating Graph Condensation: A Training-Free Approach with Class Partition

Reliable Data Distillation on Graph Convolutional Network.

On Self-Distilling Graph Neural Network

Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification

Distill to Delete: Unlearning in Graph Networks with Knowledge Distillation

NodeAug: Semi-Supervised Node Classification with Data Augmentation

Graph-based Knowledge Distillation: A survey and experimental evaluation

DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs

Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks