Abstract:Although Large Language Models (LLMs) have demonstrated potential in processing graphs, they struggle with comprehending graphical structure information through prompts of graph description sequences, especially as the graph size increases. We attribute this challenge to the uneven memory performance of LLMs across different positions in graph description sequences, known as ''positional biases''. To address this, we propose GraphInsight, a novel framework aimed at improving LLMs' comprehension of both macro- and micro-level graphical information. GraphInsight is grounded in two key strategies: 1) placing critical graphical information in positions where LLMs exhibit stronger memory performance, and 2) investigating a lightweight external knowledge base for regions with weaker memory performance, inspired by retrieval-augmented generation (RAG). Moreover, GraphInsight explores integrating these two strategies into LLM agent processes for composite graph tasks that require multi-step reasoning. Extensive empirical studies on benchmarks with a wide range of evaluation tasks show that GraphInsight significantly outperforms all other graph description methods (e.g., prompting techniques and reordering strategies) in understanding graph structures of varying sizes.

What problem does this paper attempt to address?

This paper attempts to solve the problems encountered by large - language models (LLMs) in understanding graph structures. Although LLMs show potential in handling graph data, they face challenges in understanding the structural information of graphs through natural - language - description sequences. In particular, as the graph scale increases, this understanding ability will decline significantly, ultimately leading to "understanding collapse". The author believes that this challenge mainly stems from two factors: 1. **Difficulties in long - sequence understanding**: Graph - structure information is transmitted to LLMs through language descriptions. As the graph scale increases, these descriptions become longer and longer, challenging the LLMs' ability to understand long - sequence inputs. 2. **Position bias**: The memory performance of LLMs at different positions is unbalanced, that is, "position bias". This bias is mainly reflected in the limitations of the attention mechanism and internal memory capacity, resulting in LLMs performing poorly in the middle part of the sequence (weak - memory area) and better at the head and tail (strong - memory area). To address these problems, the author proposes a new framework named GraphInsight, aiming to improve the LLMs' ability to understand graph structures by optimizing graph - description sequences. The core strategies of GraphInsight include: 1. **Utilizing strong - memory areas**: Place key graph - structure information at positions where LLMs show strong memory capabilities. 2. **Compensating for weak - memory areas**: Build a lightweight external knowledge base to enhance the performance of LLMs in weak - memory areas, similar to the concept of retrieval - augmented generation (RAG). In addition, GraphInsight also explores integrating these two strategies into the LLMs' agent process to handle complex - graph - understanding tasks that require multi - step reasoning. The main contributions of the paper include: - Conducting a pioneering analysis of the basic problems, challenges, and high - level solutions for LLMs to understand graph structures based on natural - language graph descriptions. - Proposing the first framework, GraphInsight, for the position - bias problem of LLMs in understanding graph structures. - Introducing a new benchmark, GraphSQA, for evaluating the LLMs' graph - structure - understanding ability at the macro and micro levels. - Demonstrating the effectiveness and superiority of GraphInsight through extensive empirical research.

GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understanding

GraphInstruct: Empowering Large Language Models with Graph Understanding and Reasoning Capability

GraphLLM: Boosting Graph Reasoning Ability of Large Language Model

GPT4Graph: Can Large Language Models Understand Graph Structured Data ? an Empirical Evaluation and Benchmarking.

Exploring Graph Structure Comprehension Ability of Multimodal Large Language Models: Case Studies

InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignment

Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysis

GraphEval2000: Benchmarking and Improving Large Language Models on Graph Datasets

How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehension

Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Path

Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models

Large Language Models on Graphs: A Comprehensive Survey

GUNDAM: Aligning Large Language Models with Graph Understanding

Can Language Models Solve Graph Problems in Natural Language?

Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

Large Language Model with Graph Convolution for Recommendation

A Hierarchical Language Model For Interpretable Graph Reasoning

Think-on-Graph: Deep and Responsible Reasoning of Large Language Model with Knowledge Graph