Abstract:The Graph Convolutional Network (GCN) model and its variants are powerful graph embedding tools for facilitating classification and clustering on graphs. However, a major challenge is to reduce the complexity of layered GCNs and make them parallelizable and scalable on very large graphs -- state-of the art techniques are unable to achieve scalability without losing accuracy and efficiency. In this paper, we propose novel parallelization techniques for graph sampling-based GCNs that achieve superior scalable performance on very large graphs without compromising accuracy. Specifically, our GCN guarantees work-efficient training and produces order of magnitude savings in computation and communication. To scale GCN training on tightly-coupled shared memory systems, we develop parallelization strategies for the key steps in training: For the graph sampling step, we exploit parallelism within and across multiple sampling instances, and devise an efficient data structure for concurrent accesses that provides theoretical guarantee of near-linear speedup with number of processing units. For the feature propagation step within the sampled graph, we improve cache utilization and reduce DRAM communication by data partitioning. We prove that our partitioning strategy is a 2-approximation for minimizing the communication time compared to the optimal strategy. We demonstrate that our parallel graph embedding outperforms state-of-the-art methods in scalability (with respect to number of processors, graph size and GCN model size), efficiency and accuracy on several large datasets. On a 40-core Xeon platform, our parallel training achieves $64\times$ speedup (with AVX) in the sampling step and $25\times$ speedup in the feature propagation step, compared to the serial implementation, resulting in a net speedup of $21\times$.

GraphScale: A Framework to Enable Machine Learning over Billion-node Graphs

ByteGNN: Efficient Graph Neural Network Training at Large Scale

Scalable and Efficient Full-Graph GNN Training for Large Graphs

DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs

Efficient scaling of dynamic graph neural networks

Scalable Graph Neural Networks Via Bidirectional Propagation.

GraphTheta: A Distributed Graph Neural Network Learning System With Flexible Training Strategy

GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelism

CATGNN: Cost-Efficient and Scalable Distributed Training for Graph Neural Networks

HyScale-GNN: A Scalable Hybrid GNN Training System on Single-Node Heterogeneous Architecture

Accurate, Efficient and Scalable Graph Embedding

A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking

Scaling Up Graph Neural Networks Via Graph Coarsening

Towards Efficient Large-Scale Graph Neural Network Computing.

Efficient Graph Neural Network Inference at Large Scale

Distributed Matrix-Based Sampling for Graph Neural Network Training

MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs

Haste Makes Waste: A Simple Approach for Scaling Graph Neural Networks

Accelerating GNN Training by Adapting Large Graphs to Distributed Heterogeneous Architectures

Graph Batch Coarsening Framework for Scalable Graph Neural Networks

Sketch-GNN: Scalable Graph Neural Networks with Sublinear Training Complexity