Abstract:Graph Neural Networks (GNNs) have emerged as powerful tools for supervised machine learning over graph-structured data, while sampling-based node representation learning is widely utilized in unsupervised learning. However, scalability remains a major challenge in both supervised and unsupervised learning for large graphs (e.g., those with over 1 billion nodes). The scalability bottleneck largely stems from the mini-batch sampling phase in GNNs and the random walk sampling phase in unsupervised methods. These processes often require storing features or embeddings in memory. In the context of distributed training, they require frequent, inefficient random access to data stored across different workers. Such repeated inter-worker communication for each mini-batch leads to high communication overhead and computational inefficiency. We propose GraphScale, a unified framework for both supervised and unsupervised learning to store and process large graph data distributedly. The key insight in our design is the separation of workers who store data and those who perform the training. This separation allows us to decouple computing and storage in graph training, thus effectively building a pipeline where data fetching and data computation can overlap asynchronously. Our experiments show that GraphScale outperforms state-of-the-art methods for distributed training of both GNNs and node embeddings. We evaluate GraphScale both on public and proprietary graph datasets and observe a reduction of at least 40% in end-to-end training times compared to popular distributed frameworks, without any loss in performance. While most existing methods don't support billion-node graphs for training node embeddings, GraphScale is currently deployed in production at TikTok enabling efficient learning over such large graphs.

Scalable and Efficient Full-Graph GNN Training for Large Graphs

ByteGNN: Efficient Graph Neural Network Training at Large Scale

DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs

Accelerating GNN Training by Adapting Large Graphs to Distributed Heterogeneous Architectures

BatchGNN: Efficient CPU-Based Distributed GNN Training on Very Large Graphs

CATGNN: Cost-Efficient and Scalable Distributed Training for Graph Neural Networks

NeutronTP: Load-Balanced Distributed Full-Graph GNN Training with Tensor Parallelism

Scalable Graph Neural Networks Via Bidirectional Propagation.

HyScale-GNN: A Scalable Hybrid GNN Training System on Single-Node Heterogeneous Architecture

Optimizing Task Placement and Online Scheduling for Distributed GNN Training Acceleration

GraphScale: A Framework to Enable Machine Learning over Billion-node Graphs

GraphTheta: A Distributed Graph Neural Network Learning System With Flexible Training Strategy

Towards Efficient Large-Scale Graph Neural Network Computing.

POSTER: ParGNN: Efficient Training for Large-Scale Graph Neural Network on GPU Clusters

Efficient Graph Neural Network Inference at Large Scale

Graph Neural Network Training with Data Tiering

MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs

GraphTensor: Comprehensive GNN-Acceleration Framework for Efficient Parallel Processing of Massive Datasets

A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking

Distributed Graph Neural Network Training: A Survey