Abstract:Foundation models aim to create general, cross-task, and cross-domain machine learning models by pretraining on large-scale datasets to capture shared patterns or concepts (generalities), such as contours, colors, textures, and edges in images, or tokens, words, and sentences in text. However, discovering generalities across graphs remains challenging, which has hindered the development of graph foundation models. To tackle this challenge, in this paper, we propose a novel approach to learn generalities across graphs via task-trees. Specifically, we first define the basic learning instances in graphs as task-trees and assume that the generalities shared across graphs are, at least partially, preserved in the task-trees of the given graphs. To validate the assumption, we first perform a theoretical analysis of task-trees in terms of stability, transferability, and generalization. We find that if a graph neural network (GNN) model is pretrained on diverse task-trees through a reconstruction task, it can learn sufficient transferable knowledge for downstream tasks using an appropriate set of fine-tuning samples. To empirically validate the assumption, we further instantiate the theorems by developing a cross-task, cross-domain graph foundation model named Graph generality Identifier on task-Trees (GIT). The extensive experiments over 30 graphs from five domains demonstrate the effectiveness of GIT in fine-tuning, in-context learning, and zero-shot learning scenarios. Particularly, the general GIT model pretrained on large-scale datasets can be quickly adapted to specific domains, matching or even surpassing expert models designed for those domains. Our data and code are available at <a class="link-external link-https" href="https://github.com/Zehong-Wang/GIT" rel="external noopener nofollow">this https URL</a>.

Learning Parameterized Task Structure for Generalization to Unseen Entities

GTLMA: Generalizable Hierarchical Learning for Tasks with Variable Entities

Generalizing to New Tasks via One-Shot Compositional Subgoals

Generalized Inverse Planning: Learning Lifted non-Markovian Utility for Generalizable Task Representation

Generalization of Compositional Tasks with Logical Specification via Implicit Planning

Substructure-Preserving Generalization for On-Site Robotic Construction Planning

Decompose a Task into Generalizable Subtasks in Multi-Agent Reinforcement Learning.

Robust Subtask Learning for Compositional Generalization

PIE: a Parameter and Inference Efficient Solution for Large Scale Knowledge Graph Embedding Reasoning

Task Difficulty Aware Parameter Allocation & Regularization for Lifelong Learning

Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning

Generalizable Task Planning Through Representation Pretraining

Multimodal Subtask Graph Generation from Instructional Videos

Learning from Semantically Dependent Multi-Tasks

Scoring-Aggregating-Planning: Learning Task-Agnostic Priors from Interactions and Sparse Rewards for Zero-Shot Generalization

Gradient-based inference of abstract task representations for generalization in neural networks

Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement

Synthesizing Programmatic Policy for Generalization Within Task Domain

Learning Sparse Sharing Architectures for Multiple Tasks.

Task Aware Dreamer for Task Generalization in Reinforcement Learning

Learning Cross-Task Generalities Across Graphs via Task-trees