Abstract:Despite the promising representation learning of graph neural networks (GNNs), the supervised training of GNNs notoriously requires large amounts of labeled data from each application. An effective solution is to apply the transfer learning in graph: using easily accessible information to pre-train GNNs, and fine-tuning them to optimize the downstream task with only a few labels. Recently, many efforts have been paid to design the self-supervised pretext tasks, and encode the universal graph knowledge among the various applications. However, they rarely notice the inherent training objective gap between the pretext and downstream tasks. This significant gap often requires costly fine-tuning for adapting the pre-trained model to downstream problem, which prevents the efficient elicitation of pre-trained knowledge and then results in poor results. Even worse, the naive pre-training strategy usually deteriorates the downstream task, and damages the reliability of transfer learning in graph data. To bridge the task gap, we propose a novel transfer learning paradigm to generalize GNNs, namely graph pre-training and prompt tuning (GPPT). Specifically, we first adopt the masked edge prediction, the most simplest and popular pretext task, to pre-train GNNs. Based on the pre-trained model, we propose the graph prompting function to modify the standalone node into a token pair, and reformulate the downstream node classification looking the same as edge prediction. The token pair is consisted of candidate label class and node entity. Therefore, the pre-trained GNNs could be applied without tedious fine-tuning to evaluate the linking probability of token pair, and produce the node classification decision. The extensive experiments on eight benchmark datasets demonstrate the superiority of GPPT, delivering an average improvement of 4.29% in few-shot graph analysis and accelerating the model convergence up to 4.32X. The code is available in: https://github.com/MingChen-Sun/GPPT.

Learning Cross-Task Generalities Across Graphs via Task-trees

GFT: Graph Foundation Model with Transferable Tree Vocabulary

One Model for One Graph: A New Perspective for Pretraining with Cross-domain Graphs

UniGraph: Learning a Unified Cross-Domain Foundation Model for Text-Attributed Graphs

Towards Foundation Models on Graphs: An Analysis on Cross-Dataset Transfer of Pretrained GNNs

GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learning

Generalizing Graph Transformers Across Diverse Graphs and Tasks via Pre-Training on Industrial-Scale Data

Text-Free Multi-domain Graph Pre-training: Toward Graph Foundation Models

Neural Graph Matching for Pre-training Graph Neural Networks

All in One and One for All: A Simple yet Effective Method towards Cross-domain Graph Pretraining

GPPT: Graph Pre-training and Prompt Tuning to Generalize Graph Neural Networks

Fine-tuning Graph Neural Networks by Preserving Graph Generative Patterns

Adaptive Transfer Learning on Graph Neural Networks

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training

Lecture-style Tutorial: Towards Graph Foundation Models

GraphGPT: Graph Instruction Tuning for Large Language Models

Pre-training on Large-Scale Heterogeneous Graph

Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications

GraphCLIP: Enhancing Transferability in Graph Foundation Models for Text-Attributed Graphs

Towards generalizable Graph Contrastive Learning: An information theory perspective

Graph Learning in the Era of LLMs: A Survey from the Perspective of Data, Models, and Tasks