Abstract:Graphs are omnipresent and GNNs are a powerful family of neural networks for learning over graphs. Despite their popularity, scaling GNNs either by deepening or widening suffers from prevalent issues of unhealthy gradients, over-smoothening, information squashing, which often lead to sub-standard performance. In this work, we are interested in exploring a principled way to scale GNNs capacity without deepening or widening, which can improve its performance across multiple small and large graphs. Motivated by the recent intriguing phenomenon of model soups, which suggest that fine-tuned weights of multiple large-language pre-trained models can be merged to a better minima, we argue to exploit the fundamentals of model soups to mitigate the aforementioned issues of memory bottleneck and trainability during GNNs scaling. More specifically, we propose not to deepen or widen current GNNs, but instead present a data-centric perspective of model soups tailored for GNNs, i.e., to build powerful GNNs. By dividing giant graph data, we build multiple independently and parallelly trained weaker GNNs (soup ingredient) without any intermediate communication, and combine their strength using a greedy interpolation soup procedure to achieve state-of-the-art performance. Compared to concurrent distributed GNN training works such as Jiong et. al. 2023, we train each soup ingredient by sampling different subgraphs per epoch and their respective sub-models are merged only after being fully trained (rather than intermediately so). Moreover, we provide a wide variety of model soup preparation techniques by leveraging state-of-the-art graph sampling and graph partitioning approaches that can handle large graphs. Codes are available at: \url{<a class="link-external link-https" href="https://github.com/VITA-Group/graph_ladling" rel="external noopener nofollow">this https URL</a>}.

HopGNN: Boosting Distributed GNN Training Efficiency via Feature-Centric Model Migration

Accelerating GNN Training by Adapting Large Graphs to Distributed Heterogeneous Architectures

Scalable and Efficient Full-Graph GNN Training for Large Graphs

MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs

ByteGNN: Efficient Graph Neural Network Training at Large Scale

Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communication

Expediting Distributed GNN Training with Feature-only Partition and Optimized Communication Planning

FlexGraph: a flexible and efficient distributed framework for GNN training

Optimizing Task Placement and Online Scheduling for Distributed GNN Training Acceleration

Communication-Free Distributed GNN Training with Vertex Cut

NeutronTP: Load-Balanced Distributed Full-Graph GNN Training with Tensor Parallelism

BatchGNN: Efficient CPU-Based Distributed GNN Training on Very Large Graphs

Simplifying Distributed Neural Network Training on Massive Graphs: Randomized Partitions Improve Model Aggregation

FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Training

SCGraph: Accelerating Sample-based GNN Training by Staged Caching of Features on GPUs.

BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing

DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs

Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication

Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Training

Distributed Graph Neural Network Training: A Survey