Multi-hop Question Answering over Knowledge Graphs using Large Language Models

Abir Chakraborty

2024-04-30

Abstract:Knowledge graphs (KGs) are large datasets with specific structures representing large knowledge bases (KB) where each node represents a key entity and relations amongst them are typed edges. Natural language queries formed to extract information from a KB entail starting from specific nodes and reasoning over multiple edges of the corresponding KG to arrive at the correct set of answer nodes. Traditional approaches of question answering on KG are based on (a) semantic parsing (SP), where a logical form (e.g., S-expression, SPARQL query, etc.) is generated using node and edge embeddings and then reasoning over these representations or tuning language models to generate the final answer directly, or (b) information-retrieval based that works by extracting entities and relations sequentially. In this work, we evaluate the capability of (LLMs) to answer questions over KG that involve multiple hops. We show that depending upon the size and nature of the KG we need different approaches to extract and feed the relevant information to an LLM since every LLM comes with a fixed context window. We evaluate our approach on six KGs with and without the availability of example-specific sub-graphs and show that both the IR and SP-based methods can be adopted by LLMs resulting in an extremely competitive performance.

Artificial Intelligence,Computation and Language,Databases

What problem does this paper attempt to address?

This paper discusses the use of large-scale language models (LLMs) to answer multi-hop knowledge graph (KG) questions. Traditional methods include semantic parsing (SP) and information retrieval (IR), but these methods may be inefficient in dealing with complex queries. In the paper, the authors evaluate whether LLMs can effectively answer multi-hop questions without needing a complete knowledge graph, but only partial information.

The research found that different methods are required to extract and input relevant information into LLMs based on the size and characteristics of the KG, as each LLM has a fixed-size context window. The authors tested this approach on six different KGs and found that both IR and SP methods can be adopted by LLMs, and they perform well.

Furthermore, the paper points out that for IR methods, entity or relationship lists need to be passed, while for SP methods, only linearized schema descriptions of the graph database can be passed due to the inability to directly access graph data. Additionally, the paper discusses how to dynamically extract relevant subgraphs based on the size of the KG, as well as the limitations of LLMs, such as generating misinformation and lacking domain-specific knowledge.

In conclusion, this paper addresses how to effectively utilize LLMs for handling multi-hop question answering in knowledge graphs, and proposes selecting appropriate methods based on the characteristics of different KGs, demonstrating the potential and challenges of LLMs in such tasks.