Abstract:Representation learning on text-attributed graphs (TAGs) has become a critical research problem in recent years. A typical example of a TAG is a paper citation graph, where the text of each paper serves as node attributes. Initial graph neural network (GNN) pipelines handled these text attributes by transforming them into shallow or hand-crafted features, such as skip-gram or bag-of-words features. Recent efforts have focused on enhancing these pipelines with language models (LMs), which typically demand intricate designs and substantial computational resources. With the advent of powerful large language models (LLMs) such as GPT or Llama2, which demonstrate an ability to reason and to utilize general knowledge, there is a growing need for techniques which combine the textual modelling abilities of LLMs with the structural learning capabilities of GNNs. Hence, in this work, we focus on leveraging LLMs to capture textual information as features, which can be used to boost GNN performance on downstream tasks. A key innovation is our use of explanations as features: we prompt an LLM to perform zero-shot classification, request textual explanations for its decision-making process, and design an LLM-to-LM interpreter to translate these explanations into informative features for downstream GNNs. Our experiments demonstrate that our method achieves state-of-the-art results on well-established TAG datasets, including Cora, PubMed, ogbn-arxiv, as well as our newly introduced dataset, tape-arxiv23. Furthermore, our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv. Lastly, we believe the versatility of the proposed method extends beyond TAGs and holds the potential to enhance other tasks involving graph-text data. Our codes and datasets are available at: <a class="link-external link-https" href="https://github.com/XiaoxinHe/TAPE" rel="external noopener nofollow">this https URL</a>.

STAGE: Simplified Text-Attributed Graph Embeddings Using Pre-trained LLMs

NGAT: Attention in Breadth and Depth Exploration for Semi-Supervised Graph Representation Learning

Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning

Large Language Model-based Augmentation for Imbalanced Node Classification on Text-Attributed Graphs

Disentangled Representation Learning with Large Language Models for Text-Attributed Graphs

SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning

Large Language Models as Topological Structure Enhancers for Text-Attributed Graphs

GAugLLM: Improving Graph Contrastive Learning for Text-Attributed Graphs with Large Language Models

LATEX-GCL: Large Language Models (LLMs)-Based Data Augmentation for Text-Attributed Graph Contrastive Learning

Can GNN be Good Adapter for LLMs?

Efficient and effective training of language and graph neural network models

Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs

A Pure Transformer Pretraining Framework on Text-attributed Graphs

Test-Time Training on Graphs with Large Language Models (LLMs)

Leveraging Large Language Models for Node Generation in Few-Shot Learning on Text-Attributed Graphs

Prompt-based Node Feature Extractor for Few-shot Learning on Text-Attributed Graphs

Distilling Large Language Models for Text-Attributed Graph Learning

NT-LLM: A Novel Node Tokenizer for Integrating Graph Structure into Large Language Models

Efficient End-to-end Language Model Fine-tuning on Graphs

WalkLM: A Uniform Language Model Fine-tuning Framework for Attributed Graph Embedding

Linguistic Steganalysis with Graph Neural Networks