Abstract:The collaborative reasoning for understanding each image-question pair is very critical but under-explored for an interpretable Visual Question Answering (VQA) system. Although very recent works also tried the explicit compositional processes to assemble multiple sub-tasks embedded in the questions, their models heavily rely on the annotations or hand-crafted rules to obtain valid reasoning layout, leading to either heavy labor or poor performance on composition reasoning. In this paper, to enable global context reasoning for better aligning image and language domains in diverse and unrestricted cases, we propose a novel reasoning network called Adversarial Composition Modular Network (ACMN). This network comprises of two collaborative modules: i) an adversarial attention module to exploit the local visual evidence for each word parsed from the question; ii) a residual composition module to compose the previously mined evidence. Given a dependency parse tree for each question, the adversarial attention module progressively discovers salient regions of one word by densely combining regions of child word nodes in an adversarial manner. Then residual composition module merges the hidden representations of an arbitrary number of children through sum pooling and residual connection. Our ACMN is thus capable of building an interpretable VQA system that gradually dives the image cues following a question-driven reasoning route and makes global reasoning by incorporating the learned knowledge of all attention modules in a principled manner. Experiments on relational datasets demonstrate the superiority of our ACMN and visualization results show the explainable capability of our reasoning system.

Editorial Paper for Pattern Recognition Letters VSI on Cross Model Understanding for Visual Question Answering

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Cross-modal Relational Reasoning Network for Visual Question Answering.

Explicit Cross-Modal Representation Learning for Visual Commonsense Reasoning

Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions

Joint Answering and Explanation for Visual Commonsense Reasoning

Hierarchical Multi-Modality Graph Reasoning for Remote Sensing Visual Question Answering

Graph-Structured Referring Expression Reasoning in the Wild

Visual Question Answering by Pattern Matching and Reasoning

Visual Question Reasoning on General Dependency Tree

Hierarchical Semantic Enhanced Directional Graph Network for Visual Commonsense Reasoning

Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension

Prior Visual Relationship Reasoning For Visual Question Answering

Cross-Modal Relationship Inference for Grounding Referring Expressions

Learning to Agree on Vision Attention for Visual Commonsense Reasoning

An effective spatial relational reasoning networks for visual question answering

Improving Vision-and-Language Reasoning via Spatial Relations Modeling

Modular Graph Attention Network for Complex Visual Relational Reasoning.

CMIRNet: Cross-Modal Interactive Reasoning Network for Referring Image Segmentation