Abstract:Textbook Question Answering (TQA) is a complex multimodal task to infer answers given large context descriptions and abundant diagrams. Compared with Visual Question Answering (VQA), TQA contains a large number of uncommon terminologies and various diagram inputs. It brings new challenges to the representation capability of language model for domain-specific spans. Also, it requires the model to take fully advantage of the complementary information of different diagram types, which pushes the multimodal fusion task to a more complex level. To tackle the above issues, we propose a novel model named MoCA, which incorporates M ulti-stage d o main pretraining and C ross-guided multimodal A ttention for the TQA task. Firstly, we introduce a multi-stage domain pretraining module to conduct unsupervised post-pretraining with a span mask strategy and supervised pre-finetune. Especially for domain post-pretraining, we propose a heuristic generation algorithm to employ the terminology corpus. Secondly, to fully consider the rich inputs of context and diagrams, we propose a cross-guided multimodal attention mechanism to update the features of text, question diagram and instructional diagram based on a progressive strategy. Further, a dual gating mechanism is adopted to improve the model ensemble of three background retrievals. The experimental results show the superiority of our model, which outperforms the state-of-the-art methods on the validation and test split respectively. Also, ablation and comparison experiments verify the effectiveness of each module proposed in our model.

Mnemonic reader: Machine comprehension with iterative aligning and multi-hop answer pointing

Reinforced Mnemonic Reader for Machine Comprehension

Reinforced Mnemonic Reader for Machine Reading Comprehension

MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension.

Multi-Matching Network for Multiple Choice Reading Comprehension

Machine Comprehension Using Match-LSTM and Answer Pointer

Hybrid Embedding and Joint Training of Stacked Encoder for Opinion Question Machine Reading Comprehension.

Smarnet: Teaching Machines to Read and Comprehend Like Human.

Keyword-based Query Comprehending Via Multiple Optimized-Demand Augmentation.

MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension

Iterative Alternating Neural Attention for Machine Reading

A Multi-Task Learning Machine Reading Comprehension Model for Noisy Document (student Abstract)

Learn a prior question-aware feature for machine reading comprehension

Multi-Passage Machine Reading Comprehension with Cross-Passage Answer Verification.

MoCA: Incorporating domain pretraining and cross attention for textbook question answering

MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering

EA Reader: Enhance Attentive Reader for Cloze-Style Question Answering Via Multi-Space Context Fusion.

Frame-based Neural Network for Machine Reading Comprehension

Question-Aware Memory Network for Multi-hop Question Answering in Human-Robot Interaction

DCMN+: Dual Co-Matching Network for Multi-choice Reading Comprehension

Hierarchical Answer Selection Framework For Multi-Passage Machine Reading Comprehension