Abstract:Knowledge-based question answering (KBQA) is a key task in NLP research, and also an approach to access the web data and knowledge, which requires exploiting knowledge graphs (KGs) for reasoning. In the literature, one promising solution for KBQA is to incorporate the pretrained language model (LM) with KGs by generating KG-centered pretraining corpus, which has shown its superiority. However, these methods often depend on specific techniques and resources to work, which may not always be available and restrict its application. Moreover, existing methods focus more on improving language understanding with KGs, while neglect the more important human-like complex reasoning. To this end, in this paper, we propose a general Knowledge-Injected Curriculum Pretraining framework (KICP) to achieve comprehensive KG learning and exploitation for KBQA tasks, which is composed of knowledge injection (KI), knowledge adaptation (KA) and curriculum reasoning (CR). Specifically, the KI module first injects knowledge into the LM by generating KG-centered pretraining corpus, and generalizes the process into three key steps that could work with different implementations for flexible application. Next, the KA module learns knowledge from the generated corpus with LM equipped with an adapter as well as keeps its original natural language understanding ability to reduce the negative impacts of the difference between the generated and natural corpus. Last, to enable the LM with complex reasoning, the CR module follows human reasoning patterns to construct three corpora with increasing difficulties of reasoning, and further trains the LM from easy to hard in a curriculum manner. We provide an implementation of the general framework, and evaluate the proposed KICP on four real-word datasets. The results demonstrate that our framework can achieve higher performances.

Teaching Pretrained Models with Commonsense Reasoning: A Preliminary KB-Based Approach

Evaluating Commonsense in Pre-trained Language Models

Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models

Improving Question Answering by Commonsense-Based Pre-training

Commonsense Knowledge Transfer for Pre-trained Language Models

Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge

A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation

Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey

Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations

Commonsense Reasoning for Conversational AI: A Survey of the State of the Art

CommonsenseVIS: Visualizing and Understanding Commonsense Reasoning Capabilities of Natural Language Models

A Knowledge-Injected Curriculum Pretraining Framework for Question Answering

Analyzing Commonsense Emergence in Few-shot Knowledge Models

KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for visual commonsense reasoning

Commonsense Reasoning in a Deeper Way: By Discovering Relations between Predicates.

On Commonsense Cues in BERT for Solving Commonsense Tasks.

Go Beyond Plain Fine-tuning: Improving Pretrained Models for Social Commonsense

Commonsense Knowledge Construction with Concept and Pretrained Model.

Guided Knowledge Generation with Language Models for Commonsense Reasoning

CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm

Exploring Multi-hop Reasoning Process in NLU from the View of Bayesian Probability