Abstract:Question-answering tasks over structured knowledge (i.e., tables and graphs) require the ability to encode structural information. Tra-ditional pre-trained language models trained on linear-chain natural lan-guage cannot be directly applied to encode tables and graphs. The existing methods adopt the pre-trained models in such tasks by flattening structured knowledge into sequences. However, the serialization operation will lead to the loss of the structural information of knowledge. To better employ pre -trained transformers for structured knowledge representation, we propose a novel structure-aware transformer (SATrans) that injects the local-to-global structural information of the knowledge into the mask of the different self -attention layers. Specifically, in the lower self-attention layers, SATrans fo-cus on the local structural information of each knowledge token to learn a more robust representation of it. In the upper self-attention layers, SATrans further injects the global information of the structured knowledge to inte-grate the information among knowledge tokens. In this way, the SATrans can effectively learn the semantic representation and structural information from the knowledge sequence and the attention mask, respectively. We evaluate SATrans on the table fact verification task and the knowledge base question-answering task. Furthermore, we explore two methods to com-bine symbolic and linguistic reasoning for these tasks to solve the problem that the pre-trained models lack symbolic reasoning ability. The experiment results reveal that the methods consistently outperform strong baselines on the two benchmarks.

Cluster-Former: Clustering-based Sparse Transformer for Question Answering.

Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding.

Cascade transformers with dynamic attention for video question answering

Block-Skim: Efficient Question Answering for Transformer

Local-to-Global Structure-Aware Transformer for Question Answering over Structured Knowledge.

Adaptively Clustering Neighbor Elements for Image-Text Generation

Positional Attention Guided Transformer-Like Architecture for Visual Question Answering

A lightweight Transformer-based visual question answering network with Weight-Sharing Hybrid Attention

Multimodal Graph Transformer for Multimodal Question Answering

Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers

Redundancy-aware Transformer for Video Question Answering

Structural analysis of an all-purpose question answering model

Context-aware Multi-level Question Embedding Fusion for visual question answering

Question Classification with Deep Contextualized Transformer

TransKGQA: Enhanced Knowledge Graph Question Answering With Sentence Transformers

A Question-Focused Multi-Factor Attention Network for Question Answering

Block Skim Transformer for Efficient Question Answering

Shifted Chunk Transformer for Spatio-Temporal Representational Learning

QKFormer: Hierarchical Spiking Transformer using Q-K Attention

ClusterFormer: Clustering As A Universal Visual Learner

ClusTR: Exploring Efficient Self-attention via Clustering for Vision Transformers