An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question Answering

Nan Hu,Yike Wu,Guilin Qi,Dehai Min,Jiaoyan Chen,Jeff Z. Pan,Zafar Ali

2023-03-18

Abstract:Large-scale pre-trained language models (PLMs) such as BERT have recently achieved great success and become a milestone in natural language processing (NLP). It is now the consensus of the NLP community to adopt PLMs as the backbone for downstream tasks. In recent works on knowledge graph question answering (KGQA), BERT or its variants have become necessary in their KGQA models. However, there is still a lack of comprehensive research and comparison of the performance of different PLMs in KGQA. To this end, we summarize two basic KGQA frameworks based on PLMs without additional neural network modules to compare the performance of nine PLMs in terms of accuracy and efficiency. In addition, we present three benchmarks for larger-scale KGs based on the popular SimpleQuestions benchmark to investigate the scalability of PLMs. We carefully analyze the results of all PLMs-based KGQA basic frameworks on these benchmarks and two other popular datasets, WebQuestionSP and FreebaseQA, and find that knowledge distillation techniques and knowledge enhancement methods in PLMs are promising for KGQA. Furthermore, we test ChatGPT, which has drawn a great deal of attention in the NLP community, demonstrating its impressive capabilities and limitations in zero-shot KGQA. We have released the code and benchmarks to promote the use of PLMs on KGQA.

Computation and Language

What problem does this paper attempt to address?

The main focus of this paper is on the performance, efficiency, and scalability of different pre-trained language models (PLMs) in the task of knowledge graph question answering (KGQA). Specifically, the study aims to:

1. **Comprehensively evaluate the performance of various PLMs on simple knowledge graph question answering tasks**: By designing two basic frameworks based on PLMs and comparing nine different PLMs (including BERT, RoBERTa, ALBERT, etc.) under these frameworks, the study assesses their performance in terms of accuracy and efficiency.

2. **Explore the scalability of PLMs in KGQA**: To examine how PLMs perform as the size of the knowledge graph increases, the authors constructed three knowledge graphs of different scales based on the SimpleQuestions benchmark dataset and analyzed the scalability of PLMs in such tasks.

3. **Investigate the application of lightweight and knowledge-enhanced PLMs**: The paper pays special attention to lightweight PLMs (such as DistilBERT) and knowledge-enhanced PLMs (such as KEPLER), studying whether they are suitable for KGQA tasks.

4. **Evaluate ChatGPT's KGQA capabilities in a zero-shot setting**: Although ChatGPT has demonstrated strong capabilities in multiple natural language processing tasks, the paper also tests its performance in KGQA tasks, particularly in zero-shot learning scenarios.

In summary, the main goal of this paper is to provide guidance on selecting appropriate PLMs for KGQA tasks through a comprehensive evaluation of a series of PLMs on simple knowledge graph question answering tasks, and to explore how to optimize the performance and practicality of KGQA systems.