Abstract:Graphs are omnipresent and GNNs are a powerful family of neural networks for learning over graphs. Despite their popularity, scaling GNNs either by deepening or widening suffers from prevalent issues of unhealthy gradients, over-smoothening, information squashing, which often lead to sub-standard performance. In this work, we are interested in exploring a principled way to scale GNNs capacity without deepening or widening, which can improve its performance across multiple small and large graphs. Motivated by the recent intriguing phenomenon of model soups, which suggest that fine-tuned weights of multiple large-language pre-trained models can be merged to a better minima, we argue to exploit the fundamentals of model soups to mitigate the aforementioned issues of memory bottleneck and trainability during GNNs scaling. More specifically, we propose not to deepen or widen current GNNs, but instead present a data-centric perspective of model soups tailored for GNNs, i.e., to build powerful GNNs. By dividing giant graph data, we build multiple independently and parallelly trained weaker GNNs (soup ingredient) without any intermediate communication, and combine their strength using a greedy interpolation soup procedure to achieve state-of-the-art performance. Compared to concurrent distributed GNN training works such as Jiong et. al. 2023, we train each soup ingredient by sampling different subgraphs per epoch and their respective sub-models are merged only after being fully trained (rather than intermediately so). Moreover, we provide a wide variety of model soup preparation techniques by leveraging state-of-the-art graph sampling and graph partitioning approaches that can handle large graphs. Codes are available at: \url{<a class="link-external link-https" href="https://github.com/VITA-Group/graph_ladling" rel="external noopener nofollow">this https URL</a>}.

Scalable Multi-Source Pre-training for Graph Neural Networks

Learning to Pre-train Graph Neural Networks.

One Model for One Graph: A New Perspective for Pretraining with Cross-domain Graphs

All in One and One for All: A Simple yet Effective Method towards Cross-domain Graph Pretraining

GraphFM: A Scalable Framework for Multi-Graph Pretraining

Strategies for Pre-training Graph Neural Networks

Neural Graph Matching for Pre-training Graph Neural Networks

Adaptive Transfer Learning on Graph Neural Networks

Pre-training on Large-Scale Heterogeneous Graph

GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learning

Pre-training Graph Neural Networks Via Weighted Meta Learning

Pre-Training Graph Neural Networks for Generic Structural Feature Extraction

ULTRA-DP: Unifying Graph Pre-training with Multi-task Graph Dual Prompt.

Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks

Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication

Better with Less: Data-Active Pre-training of Graph Neural Networks

Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications

Enhanced Scalable Graph Neural Network via Knowledge Distillation

DiP-GNN: Discriminative Pre-Training of Graph Neural Networks

Contrastive Pre-Training of GNNs on Heterogeneous Graphs.

Multi-Domain Generalized Graph Meta Learning.