Abstract:Question-answering tasks over structured knowledge (i.e., tables and graphs) require the ability to encode structural information. Tra-ditional pre-trained language models trained on linear-chain natural lan-guage cannot be directly applied to encode tables and graphs. The existing methods adopt the pre-trained models in such tasks by flattening structured knowledge into sequences. However, the serialization operation will lead to the loss of the structural information of knowledge. To better employ pre -trained transformers for structured knowledge representation, we propose a novel structure-aware transformer (SATrans) that injects the local-to-global structural information of the knowledge into the mask of the different self -attention layers. Specifically, in the lower self-attention layers, SATrans fo-cus on the local structural information of each knowledge token to learn a more robust representation of it. In the upper self-attention layers, SATrans further injects the global information of the structured knowledge to inte-grate the information among knowledge tokens. In this way, the SATrans can effectively learn the semantic representation and structural information from the knowledge sequence and the attention mask, respectively. We evaluate SATrans on the table fact verification task and the knowledge base question-answering task. Furthermore, we explore two methods to com-bine symbolic and linguistic reasoning for these tasks to solve the problem that the pre-trained models lack symbolic reasoning ability. The experiment results reveal that the methods consistently outperform strong baselines on the two benchmarks.

Enhancing Pre-trained Models with Text Structure Knowledge for Question Generation

Question Answering Based Clinical Text Structuring Using Pre-trained Language Model

Local-to-Global Structure-Aware Transformer for Question Answering over Structured Knowledge.

Improving Question Generation With to the Point Context

Multi-Task Learning with Language Modeling for Question Generation

Bridging the KB-Text Gap: Leveraging Structured Knowledge-aware Pre-training for KBQA

Syntax-informed Question Answering with Heterogeneous Graph Transformer

Graph augmented sequence-to-sequence model for neural question generation

Structural analysis of an all-purpose question answering model

Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering

Parallel Fusion of Graph and Text with Semantic Enhancement for Commonsense Question Answering

Enhancing Question Generation with Commonsense Knowledge

GSQA: An End-to-End Model for Generative Spoken Question Answering

Improving Visual Question Answering with Pre-Trained Language Modeling

Path-Aware Cross-Attention Network for Question Answering

Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning

ProQA: Structural Prompt-based Pre-training for Unified Question Answering

MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering

Joint Learning of Question Answering and Question Generation

Solving a Cloze Test for Generative Commonsense Question Answering.

Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering