Abstract:Knowledge graphs (KGs) complement Large Language Models (LLMs) by providing reliable, structured, domain-specific, and up-to-date external knowledge. However, KGs and LLMs are often developed separately and must be integrated after training. We introduce Tree-of-Traversals, a novel zero-shot reasoning algorithm that enables augmentation of black-box LLMs with one or more KGs. The algorithm equips a LLM with actions for interfacing a KG and enables the LLM to perform tree search over possible thoughts and actions to find high confidence reasoning paths. We evaluate on two popular benchmark datasets. Our results show that Tree-of-Traversals significantly improves performance on question answering and KG question answering tasks. Code is available at \url{<a class="link-external link-https" href="https://github.com/amazon-science/tree-of-traversals" rel="external noopener nofollow">this https URL</a>}

What problem does this paper attempt to address?

### Problems Addressed by the Paper The paper aims to address the problem of effectively combining Knowledge Graphs (KGs) with Large Language Models (LLMs). Specifically, the paper proposes a zero-shot inference algorithm named **Tree-of-Traversals**, which enhances the integration capability of LLMs with any number of internal or external knowledge graphs without retraining or fine-tuning the LLM. ### Background and Motivation 1. **Limitations of LLMs**: - **Hallucinated Information**: LLMs sometimes generate incorrect or fabricated information. - **Lack of Domain-Specific Knowledge**: LLMs may lack knowledge in certain specific domains. - **Static Knowledge Cutoff**: The knowledge of LLMs is limited to the time of their training and cannot be updated in real-time. 2. **Advantages of Knowledge Graphs**: - **Reliable, Structured, Domain-Specific, and Up-to-Date External Knowledge**: Knowledge graphs can provide these advantages, complementing the shortcomings of LLMs. 3. **Limitations of Existing Methods**: - **Pre-training, Fine-tuning, or Post-hoc Adaptation**: These methods are either computationally expensive, require public model weights, or cannot integrate large-scale knowledge graphs in memory. - **Integration of Multiple Knowledge Graphs**: Existing methods usually do not consider how to integrate multiple knowledge graphs. ### Solution The **Tree-of-Traversals** algorithm addresses the above issues in the following ways: 1. **Zero-Shot Enhancement**: It can be directly integrated with LLMs without retraining or fine-tuning. 2. **Black-Box Access**: The algorithm only requires black-box access to the LLM, without needing to understand its internal structure. 3. **Support for Multiple Knowledge Graphs**: It can integrate with any number of internal or external knowledge graphs. 4. **Tree Search Algorithm**: It uses a tree search algorithm to select actions and thought paths, finding high-confidence inference paths. ### Experiments and Evaluation 1. **Datasets**: - **2WikiMultiHop**: Used to evaluate multi-hop reasoning tasks. - **QALD-10**: Used to evaluate multilingual knowledge graph question-answering tasks. - **MusicBrainz-x-Wikidata**: A newly created dataset requiring information extraction from multiple knowledge graphs. 2. **Baseline Methods**: - **Chain-of-Thought (CoT)**: Chain-of-thought prompting. - **ReAct**: Iteratively generates thoughts and actions to retrieve information from a text knowledge base. - **FLARe**: Iteratively generates thoughts and retrieves information from a knowledge base to correct errors. 3. **Experimental Results**: - **2WikiMultiHop** and **QALD-10**: Tree-of-Traversals outperformed all baseline methods across all models, achieving state-of-the-art performance in zero-shot settings on 2WikiMultiHop. - **MusicBrainz-x-Wikidata**: Tree-of-Traversals significantly outperformed other methods on this challenging dataset requiring multi-knowledge graph reasoning. ### Conclusion The **Tree-of-Traversals** algorithm significantly improves the performance of multi-hop reasoning and knowledge graph question-answering tasks by enhancing the integration of LLMs with knowledge graphs in a zero-shot manner. It excels particularly in handling multiple knowledge graphs and domain-specific knowledge, providing a new approach for building reliable and factual LLMs.

Tree-of-Traversals: A Zero-Shot Reasoning Algorithm for Augmenting Black-box Language Models with Knowledge Graphs

Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph

Think-on-Graph: Deep and Responsible Reasoning of Large Language Model with Knowledge Graph

Neural Axiom Network for Knowledge Graph Reasoning

Reasoning on Efficient Knowledge Paths:Knowledge Graph Guides Large Language Model for Domain Question Answering

Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models

KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph

Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questions

Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuning

Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs

Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation

Tree-KGQA: An Unsupervised Approach for Question Answering Over Knowledge Graphs

Complex Logical Reasoning over Knowledge Graphs using Large Language Models

Towards Graph Foundation Models: The Perspective of Zero-shot Reasoning on Knowledge Graphs

LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities

GIVE: Structured Reasoning with Knowledge Graph Inspired Veracity Extrapolation

Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training

Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning

An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaboration