Abstract:This paper studies the task of Visual Question Answering (VQA), which is topical in Multimedia community recently. Particularly, we explore two critical research problems existed in VQA: (1) efficiently fusing the visual and textual modalities; (2) enabling the visual reasoning ability of VQA models in answering complex questions. To address these challenging problems, a novel Question Guided Modular Routing Networks (QGMRN) has been proposed in this paper. Particularly, The QGMRN is composed of visual, textual and routing network. The visual and textual network serve as the backbones for the generic feature extractors of visual and textual modalities. QGMRN can fuse the visual and textual modalities at multiple semantic levels. Typically, the visual reasoning is facilitated by the routing network in a discrete and stochastic way by using Gumbel-Softmax trick for module selection. When the input reaches a certain modular layer, routing network newly proposed in this paper, dynamically selects a portion of modules from that layer to process the input depending on the question features generated by the textual network. It can also learn to reason by routing between the generic modules without additional supervision information or expert knowledge. Benefiting from the dynamic routing mechanism, QGMRN can outperform the previous classical VQA methods by a large margin and achieve the competitive results against the state-of-the-art methods. Furthermore, attention mechanism is integrated into our QGMRN model and thus can further boost the model performance. Empirically, extensive experiments on the CLEVR and CLEVR-Humans datasets validate the effectiveness of our proposed model, and the state-of-the-art performance has been achieved.

Learning to Reason: End-to-End Module Networks for Visual Question Answering

Neural Module Networks for Reasoning over Text

Multimodal Representations for Teacher-Guided Compositional Visual Reasoning

Robust Visual Reasoning Via Language Guided Neural Module Networks

Meta Module Network for Compositional Visual Reasoning

Weakly Supervised Neuro-Symbolic Module Networks for Numerical Reasoning

Curriculum Learning for Compositional Visual Reasoning

Toward Accurate Visual Reasoning with Dual-Path Neural Module Networks.

How Modular Should Neural Module Networks Be for Systematic Generalization?

Transformer Module Networks for Systematic Generalization in Visual Question Answering

From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering

OC-NMN: Object-centric Compositional Neural Module Network for Generative Visual Analogical Reasoning

Linguistically Driven Graph Capsule Network for Visual Question Reasoning

Graph-based Heuristic Search for Module Selection Procedure in Neural Module Network

Learning to Discretely Compose Reasoning Module Networks for Video Captioning

Long-Term Memory Networks for Question Answering

Obtaining Faithful Interpretations from Compositional Neural Networks

MoReVQA: Exploring Modular Reasoning Models for Video Question Answering

Modular Blended Attention Network for Video Question Answering

Explainable and Explicit Visual Reasoning Over Scene Graphs

Question Guided Modular Routing Networks for Visual Question Answering