Abstract:Most existing visual reasoning tasks, such as CLEVR in VQA, ignore an important factor, i.e.~transformation. They are solely defined to test how well machines understand concepts and relations within static settings, like one image. Such \textbf{state driven} visual reasoning has limitations in reflecting the ability to infer the dynamics between different states, which has shown to be equally important for human cognition in Piaget's theory. To tackle this problem, we propose a novel \textbf{transformation driven} visual reasoning (TVR) task. Given both the initial and final states, the target becomes to infer the corresponding intermediate transformation. Following this definition, a new synthetic dataset namely TRANCE is first constructed on the basis of CLEVR, including three levels of settings, i.e.~Basic (single-step transformation), Event (multi-step transformation), and View (multi-step transformation with variant views). Next, we build another real dataset called TRANCO based on COIN, to cover the loss of transformation diversity on TRANCE. Inspired by human reasoning, we propose a three-staged reasoning framework called TranNet, including observing, analyzing, and concluding, to test how recent advanced techniques perform on TVR. Experimental results show that the state-of-the-art visual reasoning models perform well on Basic, but are still far from human-level intelligence on Event, View, and TRANCO. We believe the proposed new paradigm will boost the development of machine visual reasoning. More advanced methods and new problems need to be investigated in this direction. The resource of TVR is available at \url{<a class="link-external link-https" href="https://hongxin2019.github.io/TVR/" rel="external noopener nofollow">this https URL</a>}.

R el v i t: c oncept - guided v ision t ransformer for v isual r elational r easoning

Rethinking Vision Transformer Through Human–object Interaction Detection

Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects

Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCR

Transformation Driven Visual Reasoning

Question Aware Vision Transformer for Multimodal Reasoning

Hierarchical ConViT with Attention-Based Relational Reasoner for Visual Analogical Reasoning

Visual Reasoning: from State to Transformation

VisQA: X-raying Vision and Language Reasoning in Transformers

Improving Vision-and-Language Reasoning via Spatial Relations Modeling

SAViT: Structure-Aware Vision Transformer Pruning Via Collaborative Optimization.

EL-VIT: Probing Vision Transformer with Interactive Visualization

ViTCN: Vision Transformer Contrastive Network For Reasoning

CRViT: Vision transformer advanced by causality and inductive bias for image recognition

Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objects

Joint Answering and Explanation for Visual Commonsense Reasoning

ScalableViT: Rethinking the Context-Oriented Generalization of Vision Transformer.

Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios

Global Context Vision Transformers

Visual Transformation Telling

Causal-ViT: Robust Vision Transformer by causal intervention