Abstract:Inspired by the success of foundation models in applications such as ChatGPT, as graph data has been ubiquitous, one can envision the far-reaching impacts that can be brought by Graph Foundation Models (GFMs) with broader applications in the areas such as scientific research, social network analysis, drug discovery, and e-commerce. Despite the significant progress of pre-trained graph neural networks, there haven't been GFMs that can achieve desired performance on various graph-learning-related tasks. Building GFMs may rely on a vocabulary that encodes transferable patterns shared among different tasks and domains. Unlike image and text, defining such transferable patterns for graphs remains an open question. In this paper, we aim to bridge this gap by rethinking the transferable patterns on graphs as computation trees -- i.e., tree structures derived from the message-passing process. Based on this insight, we propose a cross-task, cross-domain graph foundation model named GFT, short for Graph Foundation model with transferable Tree vocabulary. By treating computation trees as tokens within the transferable vocabulary, GFT improves model generalization and reduces the risk of negative transfer. The theoretical analyses and extensive experimental studies have demonstrated the transferability of computation trees and shown the effectiveness of GFT across diverse tasks and domains in graph learning. The open source code and data are available at <a class="link-external link-https" href="https://github.com/Zehong-Wang/GFT" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to construct Graph Foundation Models (GFMs) that can be widely applied to different tasks and domains. Although pre - trained Graph Neural Networks (GNNs) have made remarkable progress, currently there are no GFMs that can achieve the expected performance in a variety of graph - learning - related tasks. This is mainly because, compared with image and text data, defining transferable patterns applicable to graph data is still an open problem. For this reason, the paper proposes a new method, that is, redefining the transferable patterns in the graph as computation trees, namely the tree structures derived from the message - passing process. Based on this insight, the author proposes a cross - task and cross - domain graph foundation model - GFT (Graph Foundation model with transferable Tree vocabulary). ### Main contributions: 1. **Computation trees as transferable patterns**: Through observation, the paper finds that the learned embeddings of each node can be captured in the form of computation trees. Therefore, computation trees can be used as transferable patterns in graph data to construct graph foundation models. 2. **GFT model**: The GFT model improves the generalization ability of the model and reduces the risk of negative transfer by regarding computation trees as tokens in the vocabulary. The GFT model includes two stages: pre - training and fine - tuning. In the pre - training stage, general knowledge is obtained through the computation tree reconstruction task, and in the fine - tuning stage, various graph - related tasks are unified into computation tree classification tasks by using the learned tree vocabulary. 3. **Theoretical analysis and experimental verification**: The paper provides theoretical analysis to prove the transferability of computation trees and verifies the effectiveness of GFT in different tasks and domains through extensive experimental research. ### Key technical points: - **Definition of computation trees**: Computation trees are sub - tree structures derived from the message - passing process and can capture local patterns in the graph. - **Learning of tree vocabulary**: By quantifying the embedding space of computation trees, a discrete tree vocabulary is formed, thereby encapsulating basic and transferable computation tree patterns. - **Computation tree reconstruction task**: In the pre - training stage, the knowledge of computation trees is comprehensively captured through computation tree reconstruction tasks in multiple aspects. - **Computation tree classification task**: In the fine - tuning stage, various graph - related tasks are unified into computation tree classification tasks to ensure that general knowledge can be adapted to specific tasks. ### Experimental results: - **Synthetic graphs**: The experimental results show that the transfer learning performance between graphs with high computation tree similarity is better, while the influence of motif similarity is smaller. - **Real - world graphs**: On real - world graph data sets, the experimental results further verify the positive influence of computation tree similarity on transfer learning performance. In general, by introducing computation trees as transferable patterns in graph data, this paper provides a new idea and method for constructing efficient graph foundation models.

GFT: Graph Foundation Model with Transferable Tree Vocabulary

Position: Graph Foundation Models are Already Here

GTA: Graph Transformer Adapter

Lecture-style Tutorial: Towards Graph Foundation Models

Towards Graph Foundation Models: A Survey and Beyond

Learning Cross-Task Generalities Across Graphs via Task-trees

Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights

GOFA: A Generative One-For-All Model for Joint Graph Language Modeling

LangGFM: A Large Language Model Alone Can be a Powerful Graph Foundation Model

OpenGraph: Towards Open Graph Foundation Models

UniGraph: Learning a Unified Cross-Domain Foundation Model for Text-Attributed Graphs

GraphFM: A Comprehensive Benchmark for Graph Foundation Model

GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learning

Text-Free Multi-domain Graph Pre-training: Toward Graph Foundation Models

Towards Foundation Models on Graphs: An Analysis on Cross-Dataset Transfer of Pretrained GNNs

Enabling Foundation Models: A Distributed Collaboration Framework Based on Graph Federated Learning

GraphTransfer: A Generic Feature Fusion Framework for Collaborative Filtering

Training Free Graph Neural Networks for Graph Matching

Boosting Graph Foundation Model from Structural Perspective

FedGT: Federated Node Classification with Scalable Graph Transformer

Generalizing Graph Transformers Across Diverse Graphs and Tasks via Pre-Training on Industrial-Scale Data