ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling

Linyao Yang,Hongyang Chen,Zhao Li,Xiao Ding,Xindong Wu

DOI: https://doi.org/10.48550/arXiv.2306.11489

2023-06-20

Computation and Language

Abstract:Recently, ChatGPT, a representative large language model (LLM), has gained considerable attention due to its powerful emergent abilities. Some researchers suggest that LLMs could potentially replace structured knowledge bases like knowledge graphs (KGs) and function as parameterized knowledge bases. However, while LLMs are proficient at learning probabilistic language patterns based on large corpus and engaging in conversations with humans, they, like previous smaller pre-trained language models (PLMs), still have difficulty in recalling facts while generating knowledge-grounded contents. To overcome these limitations, researchers have proposed enhancing data-driven PLMs with knowledge-based KGs to incorporate explicit factual knowledge into PLMs, thus improving their performance to generate texts requiring factual knowledge and providing more informed responses to user queries. This paper reviews the studies on enhancing PLMs with KGs, detailing existing knowledge graph enhanced pre-trained language models (KGPLMs) as well as their applications. Inspired by existing studies on KGPLM, this paper proposes to enhance LLMs with KGs by developing knowledge graph-enhanced large language models (KGLLMs). KGLLM provides a solution to enhance LLMs' factual reasoning ability, opening up new avenues for LLM research.

What problem does this paper attempt to address?

The paper primarily explores how to enhance the factual reasoning capabilities of Large Language Models (LLMs) through Knowledge Graphs (KGs). Specifically, the paper points out that although current LLMs exhibit powerful emergent abilities on large-scale corpora, they still face difficulties in generating fact-based content. This is because LLMs can only remember a limited amount of factual information during training, resulting in poor performance when generating content that requires specific factual support. To address this issue, researchers propose combining structured Knowledge Graphs with LLMs, leveraging the explicit factual knowledge in KGs to improve the performance of LLMs.

The main contributions of the paper include:
1. A comprehensive review of existing Knowledge Graph Enhanced Pre-trained Language Models (KGPLMs), helping researchers gain an in-depth understanding of the research progress in this field.
2. A comparison of the differences between LLMs and KGs, and an exploration of whether LLMs can replace traditional Knowledge Graphs.
3. The proposal of a new method—Knowledge Graph Enhanced Large Language Models (KGLLMs)—to improve the performance of LLMs in tasks requiring factual reasoning, and an indication of future research directions.

In summary, the paper aims to address the inadequacy of LLMs in generating content that requires specific factual support and explores how to improve this by introducing Knowledge Graphs.