Abstract:Graph neural networks (GNNs) tend to suffer from high computation costs due to the exponentially increasing scale of graph data and a large number of model parameters, which restricts their utility in practical applications. To this end, some recent works focus on sparsifying GNNs (including graph structures and model parameters) with the lottery ticket hypothesis (LTH) to reduce inference costs while maintaining performance levels. However, the LTH-based methods suffer from two major drawbacks: 1) they require exhaustive and iterative training of dense models, resulting in an extremely large training computation cost, and 2) they only trim graph structures and model parameters but ignore the node feature dimension, where vast redundancy exists. To overcome the above limitations, we propose a comprehensive graph gradual pruning framework termed CGP. This is achieved by designing a during-training graph pruning paradigm to dynamically prune GNNs within one training process. Unlike LTH-based methods, the proposed CGP approach requires no retraining, which significantly reduces the computation costs. Furthermore, we design a cosparsifying strategy to comprehensively trim all the three core elements of GNNs: graph structures, node features, and model parameters. Next, to refine the pruning operation, we introduce a regrowth process into our CGP framework, to reestablish the pruned but important connections. The proposed CGP is evaluated over a node classification task across six GNN architectures, including shallow models [graph convolutional network (GCN) and graph attention network (GAT)], shallow-but-deep-propagation models [simple graph convolution (SGC) and approximate personalized propagation of neural predictions (APPNP)], and deep models [GCN via initial residual and identity mapping (GCNII) and residual GCN (ResGCN)], on a total of 14 real-world graph datasets, including large-scale graph datasets from the challenging Open Graph Benchmark (OGB). Experiments reveal that the proposed strategy greatly improves both training and inference efficiency while matching or even exceeding the accuracy of the existing methods.

Learning soft threshold for sparse reparameterization using gradual projection operators

Deep Neural Network Acceleration with Sparse Prediction Layers

SUBP: Soft Uniform Block Pruning for 1 X N Sparse CNNs Multithreading Acceleration

Batch-Normalization-based Soft Filter Pruning for Deep Convolutional Neural Networks

SUBP: Soft Uniform Block Pruning for 1xn Sparse CNNs Multithreading Acceleration

Joint Sparsity with Mixed Granularity for Efficient GPU Implementation

Structured Probabilistic Pruning for Convolutional Neural Network Acceleration.

Neural Network Pruning by Gradient Descent

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

FGGP: Fixed-Rate Gradient-First Gradual Pruning

Spatial-Winograd Pruning Enabling Sparse Winograd Convolution

Comprehensive Graph Gradual Pruning for Sparse Training in Graph Neural Networks

Network Pruning Using Sparse Learning and Genetic Algorithm

Effective Model Sparsification by Scheduled Grow-and-Prune Methods

Enabling Retrain-free Deep Neural Network Pruning using Surrogate Lagrangian Relaxation

PRUNING IN TRAINING: LEARNING AND RANKING SPARSE CONNECTIONS IN DEEP CONVOLUTIONAL NETWORKS

Group Pruning with Group Sparse Regularization for Deep Neural Network Compression

Global balanced iterative pruning for efficient convolutional neural networks

Efficient Neural Networks with Spatial Wise Sparsity Using Unified Importance Map.

Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning

CGaP: Continuous Growth and Pruning for Efficient Deep Learning