Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"

Saeed Amizadeh,Hamid Palangi,Oleksandr Polozov,Yichen Huang,Kazuhito Koishida

DOI: https://doi.org/10.48550/arXiv.2006.11524

2020-08-26

Abstract:Visual reasoning tasks such as visual question answering (VQA) require an interplay of visual perception with reasoning about the question semantics grounded in perception. However, recent advances in this area are still primarily driven by perception improvements (e.g. scene graph generation) rather than reasoning. Neuro-symbolic models such as Neural Module Networks bring the benefits of compositional reasoning to VQA, but they are still entangled with visual representation learning, and thus neural reasoning is hard to improve and assess on its own. To address this, we propose (1) a framework to isolate and evaluate the reasoning aspect of VQA separately from its perception, and (2) a novel top-down calibration technique that allows the model to answer reasoning questions even with imperfect perception. To this end, we introduce a differentiable first-order logic formalism for VQA that explicitly decouples question answering from visual perception. On the challenging GQA dataset, this framework is used to perform in-depth, disentangled comparisons between well-known VQA models leading to informative insights regarding the participating models as well as the task.

Machine Learning,Artificial Intelligence,Computer Vision and Pattern Recognition,Neural and Evolutionary Computing,Symbolic Computation

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the decoupling problem between perception (visual understanding) and reasoning (multi - step inference based on scene - based understanding) in visual reasoning tasks. Specifically, the paper points out that the current progress in the field of Visual Question Answering (VQA) mainly focuses on improving visual perception ability, while relatively little improvement and evaluation have been made on reasoning ability. This has led to poor performance of neural network models when dealing with problems requiring complex reasoning.

To meet this challenge, the author makes two main contributions:
1. **Framework**: A framework is proposed to separate the reasoning part from the perception part in VQA and independently evaluate its performance.
2. **Calibration technique**: A top - down calibration technique is introduced, enabling the model to answer reasoning questions even when perception is imperfect.

Through these methods, the paper aims to more accurately evaluate the reasoning ability of VQA models and provide a tool to improve the performance of these models when dealing with complex reasoning tasks.