Abstract:The Graph Convolutional Network (GCN) model and its variants are powerful graph embedding tools for facilitating classification and clustering on graphs. However, a major challenge is to reduce the complexity of layered GCNs and make them parallelizable and scalable on very large graphs -- state-of the art techniques are unable to achieve scalability without losing accuracy and efficiency. In this paper, we propose novel parallelization techniques for graph sampling-based GCNs that achieve superior scalable performance on very large graphs without compromising accuracy. Specifically, our GCN guarantees work-efficient training and produces order of magnitude savings in computation and communication. To scale GCN training on tightly-coupled shared memory systems, we develop parallelization strategies for the key steps in training: For the graph sampling step, we exploit parallelism within and across multiple sampling instances, and devise an efficient data structure for concurrent accesses that provides theoretical guarantee of near-linear speedup with number of processing units. For the feature propagation step within the sampled graph, we improve cache utilization and reduce DRAM communication by data partitioning. We prove that our partitioning strategy is a 2-approximation for minimizing the communication time compared to the optimal strategy. We demonstrate that our parallel graph embedding outperforms state-of-the-art methods in scalability (with respect to number of processors, graph size and GCN model size), efficiency and accuracy on several large datasets. On a 40-core Xeon platform, our parallel training achieves $64\times$ speedup (with AVX) in the sampling step and $25\times$ speedup in the feature propagation step, compared to the serial implementation, resulting in a net speedup of $21\times$.

HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training

HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework

Task-Oriented Genetic Activation for Large-Scale Complex Heterogeneous Graph Embedding.

GraphScale: A Framework to Enable Machine Learning over Billion-node Graphs

Accurate, Efficient and Scalable Graph Embedding

Haste Makes Waste: A Simple Approach for Scaling Graph Neural Networks

HET-KG: Communication-Efficient Knowledge Graph Embedding Training Via Hotness-Aware Cache

MILE: A Multi-Level Framework for Scalable Graph Embedding

Graph Relation Embedding Network for Click-Through Rate Prediction

Graph Embedding Via Multi-Scale Graph Representations

A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking

Generalizable Cross-Graph Embedding for GNN-based Congestion Prediction

A Distributed Multi-GPU System for Large-Scale Node Embedding at Tencent

Efficient Knowledge Graph Embedding Training Framework with Multiple GPUs

HeteroMILE: a Multi-Level Graph Representation Learning Framework for Heterogeneous Graphs

PyTorch-BigGraph: A Large-scale Graph Embedding System

Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis

LD2: Scalable Heterophilous Graph Neural Network with Decoupled Embeddings

SketchNE: Embedding Billion-Scale Networks Accurately in One Hour

Heta: Distributed Training of Heterogeneous Graph Neural Networks

Unsupervised Embedding Learning for Large-Scale Heterogeneous Networks Based on Metapath Graph Sampling