ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph

Jinhao Jiang,Kun Zhou,Wayne Xin Zhao,Yaliang Li,Ji-Rong Wen

2023-12-30

Abstract:Question Answering over Knowledge Graph (KGQA) aims to seek answer entities for the natural language question from a large-scale Knowledge Graph~(KG). To better perform reasoning on KG, recent work typically adopts a pre-trained language model~(PLM) to model the question, and a graph neural network~(GNN) based module to perform multi-hop reasoning on the KG. Despite the effectiveness, due to the divergence in model architecture, the PLM and GNN are not closely integrated, limiting the knowledge sharing and fine-grained feature interactions. To solve it, we aim to simplify the above two-module approach, and develop a more capable PLM that can directly support subgraph reasoning for KGQA, namely ReasoningLM. In our approach, we propose a subgraph-aware self-attention mechanism to imitate the GNN for performing structured reasoning, and also adopt an adaptation tuning strategy to adapt the model parameters with 20,000 subgraphs with synthesized questions. After adaptation, the PLM can be parameter-efficient fine-tuned on downstream tasks. Experiments show that ReasoningLM surpasses state-of-the-art models by a large margin, even with fewer updated parameters and less training data. Our codes and data are publicly available at~\url{

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

This paper proposes a solution to the problem of structured subgraph reasoning in Knowledge Graph Question Answering (KGQA). Existing methods usually use Pre-trained Language Models (PLMs) to process questions and perform multi-hop reasoning on the Knowledge Graph (KG) using Graph Neural Networks (GNNs). However, due to the differences in these two model architectures, knowledge sharing and fine-grained interaction between them are limited. Therefore, the paper introduces a new approach called ReasoningLM, aiming to simplify the two-module approach of PLMs and GNNs, and develop a more powerful PLM that can directly support KGQA with subgraph reasoning capability.

ReasoningLM mimics GNNs by introducing a subgraph-aware self-attention mechanism to perform reasoning in a structured manner, and combines with a constraint mask framework to handle question attention, KG-to-question attention, and KG attention simultaneously. In order to adapt PLMs to this special input format and attention mechanism, the paper proposes an adaptive fine-tuning strategy, which utilizes 20,000 subgraphs with synthetic questions to adjust the parameters of PLMs. After fine-tuning, PLMs can achieve fine-grained adjustments in downstream tasks with parameter efficiency, even with fewer updated parameters and less training data, outperforming existing state-of-the-art models.

Experiments show that ReasoningLM performs significantly better than other models on three KGQA datasets, validating its effectiveness and efficiency.