Abstract:Large language models (LLMs) are being increasingly explored for graph tasks. Despite their remarkable success in text-based tasks, LLMs' capabilities in understanding explicit graph structures remain limited, particularly with large graphs. In this work, we introduce Hierarchical Language Model for Graph (HLM-G), which employs a two-block architecture to capture node-centric local information and interaction-centric global structure, effectively enhancing graph structure understanding abilities. The proposed scheme allows LLMs to address various graph queries with high efficacy, efficiency, and robustness, while reducing computational costs on large-scale graph tasks. Furthermore, we demonstrate the interpretability of our model using intrinsic attention weights and established explainers. Comprehensive evaluations across diverse graph reasoning and real-world tasks of node, link, and graph-levels highlight the superiority of our method, marking a significant advancement in the application of LLMs to graph understanding.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to improve the performance of large - language models (LLMs) in graph - structure tasks, especially the efficiency, effectiveness and interpretability when dealing with large - scale graph data. Specifically, although existing LLMs perform well in text tasks, they have limitations in understanding and processing explicit graph structures, especially when dealing with large graphs. To solve these problems, the paper proposes a new framework named Hierarchical Language Model for Graphs (HLM - G). ### Main problems: 1. **Limited ability to understand graph structures**: - LLMs are good at processing feature - based information, but perform poorly in capturing graph - structure details. For example, in simple graph tasks such as shortest - path identification, the performance of LLMs is not as good as that of specialized graph neural networks (GNNs). - For complex link and graph - level tasks, LLMs have difficulty effectively understanding long - range structures. 2. **Scalability challenges**: - Encoding graphs into LLMs will bring significant computational complexity problems. After the feature and structure information of each node is encoded into a natural - language description, the input sequence will be too long, causing the computational cost to increase dramatically, especially for large - scale graph tasks. 3. **Lack of interpretability**: - Traditional GNNs rely on opaque embedding representations and lack human - understandable explanations. Although LLMs can enhance interpretability through natural - language descriptions, there has not been any work specifically for graph - structure explanations yet. ### Solutions: The paper proposes a hierarchical language model HLM - G, which solves the above problems in the following ways: - **Hierarchical architecture**: Adopting a two - block architecture to capture the local information centered on nodes and the global structure centered on interactions respectively. This design not only enhances the understanding of graph structures, but also reduces the computational cost and improves the scalability of the model. - **Efficient attention mechanism**: Introducing a block - diagonal attention mechanism, which reduces the computational complexity from the traditional \(O((\sum n_i)^2)\) to \(O(\sum n_i^2)\), thus improving the efficiency. - **Interpretability**: Demonstrating the interpretability of the model through the intrinsic attention weights and the established interpreter, enabling users to better understand the decision - making process of the model. In summary, this paper aims to improve the performance of LLMs in graph - structure tasks through the HLM - G framework while ensuring its efficiency and interpretability.

A Hierarchical Language Model For Interpretable Graph Reasoning

GraphLLM: Boosting Graph Reasoning Ability of Large Language Model

GUNDAM: Aligning Large Language Models with Graph Understanding

Large Language Models on Graphs: A Comprehensive Survey

Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Path

GraphInstruct: Empowering Large Language Models with Graph Understanding and Reasoning Capability

Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysis

Large Language Model with Graph Convolution for Recommendation

Beyond Graphs: Can Large Language Models Comprehend Hypergraphs?

A Survey of Large Language Models for Graphs

GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understanding

Enhancing Recommender Systems with Large Language Model Reasoning Graphs

Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs

A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applications

Can Language Models Solve Graph Problems in Natural Language?

Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models

GraphEval2000: Benchmarking and Improving Large Language Models on Graph Datasets

End-to-End Graph Flattening Method for Large Language Models

GPT4Graph: Can Large Language Models Understand Graph Structured Data ? an Empirical Evaluation and Benchmarking.

Large Language Models as Topological Structure Enhancers for Text-Attributed Graphs