Abstract:Textbook Question Answering (TQA) task requires answering questions by reasoning based on both the given diagrams and text context. There are mainly two challenges for the task. First, the diagrams are different from the natural images. Similar shapes or color blocks may express different semantics and there is also a large intra-topic variation for diagrams. Hence, the characteristics of visual semantic ambiguity and variable visual appearance make the diagram understanding more challenging. Second, for the text, the specific education domain with terminologies exists a great gap with the general domain. Therefore, it is difficult to represent the text semantics effectively using a text encoder pretrained in the general domain. In this paper, we propose a Spatial-Semantic Collaborative Graph Network (SSCGN) for TQA task, which can help enhance the diagram and text understanding and facilitate multimodal reasoning. Specifically, the Spatial-guided Semantic Enhancing (SSE) module fully exploits the spatial and semantic relationships between visual objects and OCR tokens to collaboratively enhance the diagram semantic understanding. Moreover, based on the semantically enhanced region representations of the SSE module, the Fine-grained Spatial-Aware Graph Network (FSA-GN) can help obtain richer relation-aware region representations for joint reasoning by capturing more fine-grained spatial relationships. We further propose multiple self-supervised auxiliary tasks to enhance the initial diagram and text semantic representations by pretraining the diagram encoder and text encoder. Extensive experiments and ablation studies are conducted to validate the effectiveness of SSCGN.

Parallel Fusion of Graph and Text with Semantic Enhancement for Commonsense Question Answering

Fusing Context Into Knowledge Graph for Commonsense Question Answering

FeQA: Fusion and enhancement of multi-source knowledge on question answering

Subgraph Retrieval Enhanced by Graph-Text Alignment for Commonsense Question Answering

ALBERT with Knowledge Graph Encoder Utilizing Semantic Similarity for Commonsense Question Answering

Knowledge Graph Question Answering with semantic oriented fusion model.

Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources

Coarse-to-Careful: Seeking Semantic-related Knowledge for Open-domain Commonsense Question Answering

Knowledge-aware adaptive graph network for commonsense question answering

Benchmarking Knowledge-Enhanced Commonsense Question Answering via Knowledge-to-Text Transformation

Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering

KE-GCL: Knowledge Enhanced Graph Contrastive Learning for Commonsense Question Answering.

AFS Graph: Multidimensional Axiomatic Fuzzy Set Knowledge Graph for Open-Domain Question Answering

Cross-Attentional Spatio-Temporal Semantic Graph Networks for Video Question Answering

Spatial-Semantic Collaborative Graph Network for Textbook Question Answering

CORN: Co-Reasoning Network for Commonsense Question Answering.

A Semantic-based Method for Unsupervised Commonsense Question Answering

Incorporating Domain Knowledge and Semantic Information into Language Models for Commonsense Question Answering

Video Question Answering with Semantic Disentanglement and Reasoning

Question-Driven Graph Fusion Network For Visual Question Answering

Modeling Global Semantics for Question Answering over Knowledge Bases