Abstract:Graph Neural Networks (GNNs) have emerged as powerful tools for supervised machine learning over graph-structured data, while sampling-based node representation learning is widely utilized in unsupervised learning. However, scalability remains a major challenge in both supervised and unsupervised learning for large graphs (e.g., those with over 1 billion nodes). The scalability bottleneck largely stems from the mini-batch sampling phase in GNNs and the random walk sampling phase in unsupervised methods. These processes often require storing features or embeddings in memory. In the context of distributed training, they require frequent, inefficient random access to data stored across different workers. Such repeated inter-worker communication for each mini-batch leads to high communication overhead and computational inefficiency. We propose GraphScale, a unified framework for both supervised and unsupervised learning to store and process large graph data distributedly. The key insight in our design is the separation of workers who store data and those who perform the training. This separation allows us to decouple computing and storage in graph training, thus effectively building a pipeline where data fetching and data computation can overlap asynchronously. Our experiments show that GraphScale outperforms state-of-the-art methods for distributed training of both GNNs and node embeddings. We evaluate GraphScale both on public and proprietary graph datasets and observe a reduction of at least 40% in end-to-end training times compared to popular distributed frameworks, without any loss in performance. While most existing methods don't support billion-node graphs for training node embeddings, GraphScale is currently deployed in production at TikTok enabling efficient learning over such large graphs.

A Distributed Multi-GPU System for Large-Scale Node Embedding at Tencent

GraphVite: A High-Performance CPU-GPU Hybrid System for Node Embedding

Efficient Partitioning and Communication Scheme-Based Distributed Edge Computing to Accelerate Deep Neural Network

SPEED: Streaming Partition and Parallel Acceleration for Temporal Interaction Graph Embedding

On Distributed Deep Network for Processing Large-Scale Sets of Complex Data

GraphScale: A Framework to Enable Machine Learning over Billion-node Graphs

Distributed Graph Embedding with Information-Oriented Random Walks

PyTorch-BigGraph: A Large-scale Graph Embedding System

HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Models

Legion: Automatically Pushing the Envelope of Multi-GPU System for Billion-Scale GNN Training

GraphPEG

HongTu: Scalable Full-Graph GNN Training on Multiple GPUs (via communication-optimized CPU data offloading)

Pipeline Parallelism for Inference on Heterogeneous Edge Computing

DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs

Puffin: Graph Processing System on Multi-GPUs

HyScale-GNN: A Scalable Hybrid GNN Training System on Single-Node Heterogeneous Architecture

Scalable and Efficient Full-Graph GNN Training for Large Graphs

HetHub: A Heterogeneous Distributed Hybrid Training System for Large-Scale Models

BGS: Accelerate GNN Training on Multiple GPUs

CATGNN: Cost-Efficient and Scalable Distributed Training for Graph Neural Networks

Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters