Abstract:Training graph neural networks (GNNs) on large-scale graph data holds immense promise for numerous real-world applications but remains a great challenge. Several disk-based GNN systems have been built to train large-scale graphs in a single machine. However, they often fall short in terms of performance, especially when training on terabyte-scale graphs. This is because existing disk-based systems either overly focus on minimizing the number of SSD accesses or do not fully overlap SSD accesses with GNN training, thus resulting in substantial unnecessary overhead on the CPU side and then low GPU utilization. To this end, we propose Helios, a system that can train GNN on terabyte graphs in a single machine while achieving throughput comparable with in-memory systems. To achieve this, we first present a GPU-initiated asynchronous disk IO stack, allowing the GPU to directly access graph data on SSD. This design only requires about 30% GPU cores to reach the almost maximal disk IO throughput and wastes no GPU cores between IO submission and IO completion such that the majority of GPU cores are left for other GNN kernels. Second, we design a GPU-managed heterogeneous cache that extends the cache hierarchy to heterogeneous CPU and GPU memory and thus enhances cache lookup throughput significantly by GPU parallelism. Finally, we build a deep GNN-aware pipeline that seamlessly integrates the computation and communication phases of the entire GNN training process, maximizing the utility of GPU computation cycles. Experimental results demonstrate that Helios can match the training throughput of in-memory GNN systems, even for terabyte-scale graphs. Remarkably, Helios surpasses the state-of-the-art GPU-managed baselines by up to 6.43x and exceeds CPU-managed baselines by over 182x on all terabyte-scale graphs.

MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networks

LSM-GNN: Large-scale Storage-based Multi-GPU GNN Training by Optimizing Data Transfer Scheme

ByteGNN: Efficient Graph Neural Network Training at Large Scale

CATGNN: Cost-Efficient and Scalable Distributed Training for Graph Neural Networks

DiskGNN: Bridging I/O Efficiency and Model Accuracy for Out-of-Core GNN Training

Helios: An Efficient Out-of-core GNN Training System on Terabyte-scale Graphs with In-memory Performance

Reducing Memory Contention and I/O Congestion for Disk-based GNN Training

Graph Neural Network Training with Data Tiering

Scalable and Efficient Full-Graph GNN Training for Large Graphs

BatchGNN: Efficient CPU-Based Distributed GNN Training on Very Large Graphs

GNNear: Accelerating Full-Batch Training of Graph Neural Networks with Near-Memory Processing

HyScale-GNN: A Scalable Hybrid GNN Training System on Single-Node Heterogeneous Architecture

XGNN: Boosting Multi-GPU GNN Training Via Global GNN Memory Store.

DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs

BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing

DUCATI: A Dual-Cache Training System for Graph Neural Networks on Giant Graphs with the GPU.

Efficient scaling of dynamic graph neural networks

MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs

ARGO: An Auto-Tuning Runtime System for Scalable GNN Training on Multi-Core Processor

Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture