Abstract:Textbook Question Answering (TQA) task requires answering questions by reasoning based on both the given diagrams and text context. There are mainly two challenges for the task. First, the diagrams are different from the natural images. Similar shapes or color blocks may express different semantics and there is also a large intra-topic variation for diagrams. Hence, the characteristics of visual semantic ambiguity and variable visual appearance make the diagram understanding more challenging. Second, for the text, the specific education domain with terminologies exists a great gap with the general domain. Therefore, it is difficult to represent the text semantics effectively using a text encoder pretrained in the general domain. In this paper, we propose a Spatial-Semantic Collaborative Graph Network (SSCGN) for TQA task, which can help enhance the diagram and text understanding and facilitate multimodal reasoning. Specifically, the Spatial-guided Semantic Enhancing (SSE) module fully exploits the spatial and semantic relationships between visual objects and OCR tokens to collaboratively enhance the diagram semantic understanding. Moreover, based on the semantically enhanced region representations of the SSE module, the Fine-grained Spatial-Aware Graph Network (FSA-GN) can help obtain richer relation-aware region representations for joint reasoning by capturing more fine-grained spatial relationships. We further propose multiple self-supervised auxiliary tasks to enhance the initial diagram and text semantic representations by pretraining the diagram encoder and text encoder. Extensive experiments and ablation studies are conducted to validate the effectiveness of SSCGN.

A Synergistic Framework for Geographic Question Answering

Geo-analytical question-answering with GIS

Geographic Question Answering: Challenges, Uniqueness, Classification, and Future Directions

A Question-Answering System over Traditional Chinese Medicine

Semantic Computation in Geography Question Answering

Relaxing Unanswerable Geographic Questions Using A Spatially Explicit Knowledge Graph Embedding Model

Semantic Web Enabled Geographic Question Answering Framework: GeoTR

A Chinese Intelligent Question Answering System Based on Domain Ontology and Sentence Templates

Spatial Relation Query Based on Geographic Ontology

GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level

Spatial-Semantic Collaborative Graph Network for Textbook Question Answering

Semantic information integration and question answering based on pervasive agent ontology

A Survey on Frameworks and Methods of Question Answering

Towards a Question Answering System over the Semantic Web

A Question Answering System Built On Domain Knowledge Base

Question answering based on pervasive agent ontology and Semantic Web

A Chinese Question Answering System For Specific Domain

Probabilistic qualitative spatial reasoning with applications to GeoQA

Open Domain Question Answering Via Semantic Enrichment

Towards Answering Geography Questions in Gaokao: A Hybrid Approach

ComQA: Question Answering over Knowledge Base Via Semantic Matching.