Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning

Zhengyu Hu,Yichuan Li,Zhengyu Chen,Jingang Wang,Han Liu,Kyumin Lee,Kaize Ding

2024-10-10

Abstract:Textual Attributed Graphs (TAGs) are crucial for modeling complex real-world systems, yet leveraging large language models (LLMs) for TAGs presents unique challenges due to the gap between sequential text processing and graph-structured data. We introduce AskGNN, a novel approach that bridges this gap by leveraging In-Context Learning (ICL) to integrate graph data and task-specific information into LLMs. AskGNN employs a Graph Neural Network (GNN)-powered structure-enhanced retriever to select labeled nodes across graphs, incorporating complex graph structures and their supervision signals. Our learning-to-retrieve algorithm optimizes the retriever to select example nodes that maximize LLM performance on graph. Experiments across three tasks and seven LLMs demonstrate AskGNN's superior effectiveness in graph task performance, opening new avenues for applying LLMs to graph-structured data without extensive fine-tuning.

Machine Learning

What problem does this paper attempt to address?

The problem this paper attempts to address is how to effectively apply large language models (LLMs) to tasks involving text attribute graphs (TAGs). Although LLMs perform excellently on many tasks, they have significant limitations in processing and utilizing the structural information in TAGs. This is because LLMs are primarily trained on unstructured text data and lack the ability to interpret and utilize graph structural information. This limits their effectiveness in tasks that require a deep understanding of the interrelationships between text documents. To overcome this limitation, the paper proposes a new framework called AskGNN. AskGNN enhances the performance of LLMs on graph tasks without extensive fine-tuning by introducing a graph neural network (GNN)-enhanced retriever that transforms graph structural information and task supervision signals into a small number of document node label pairs. Specifically, the core of AskGNN is a GNN-based structure-enhanced retriever that can select the most relevant document node label pairs as context examples for LLMs to support their few-shot learning capability. In summary, this paper aims to bridge the gap in structural information processing for large language models when dealing with text attribute graphs through the AskGNN framework, thereby improving the performance of LLMs on graph tasks.

Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning

Disentangled Representation Learning with Large Language Models for Text-Attributed Graphs

Can GNN be Good Adapter for LLMs?

Large Language Models as Topological Structure Enhancers for Text-Attributed Graphs

All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networks

LLaGA: Large Language and Graph Assistant

Graph Neural Prompting with Large Language Models

Distilling Large Language Models for Text-Attributed Graph Learning

LOGIN: A Large Language Model Consulted Graph Neural Network Training Framework

Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs

Large Language Models on Graphs: A Comprehensive Survey

GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning

Can Large Language Models Act as Ensembler for Multi-GNNs?

Efficient and effective training of language and graph neural network models

Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?

Learning on Large-scale Text-attributed Graphs via Variational Inference

A Survey of Large Language Models for Graphs

GNN-LM: Language Modeling based on Global Contexts via GNN

Can Language Models Solve Graph Problems in Natural Language?

NT-LLM: A Novel Node Tokenizer for Integrating Graph Structure into Large Language Models