Abstract:The Abstraction and Reasoning Corpus (ARC) is a popular benchmark focused on visual reasoning in the evaluation of Artificial Intelligence systems. In its original framing, an ARC task requires solving a program synthesis problem over small 2D images using a few input-output training pairs. In this work, we adopt the recently popular data-driven approach to the ARC and ask whether a Vision Transformer (ViT) can learn the implicit mapping, from input image to output image, that underlies the task. We show that a ViT -- otherwise a state-of-the-art model for images -- fails dramatically on most ARC tasks even when trained on one million examples per task. This points to an inherent representational deficiency of the ViT architecture that makes it incapable of uncovering the simple structured mappings underlying the ARC tasks. Building on these insights, we propose ViTARC, a ViT-style architecture that unlocks some of the visual reasoning capabilities required by the ARC. Specifically, we use a pixel-level input representation, design a spatially-aware tokenization scheme, and introduce a novel object-based positional encoding that leverages automatic segmentation, among other enhancements. Our task-specific ViTARC models achieve a test solve rate close to 100% on more than half of the 400 public ARC tasks strictly through supervised learning from input-output grids. This calls attention to the importance of imbuing the powerful (Vision) Transformer with the correct inductive biases for abstract visual reasoning that are critical even when the training data is plentiful and the mapping is noise-free. Hence, ViTARC provides a strong foundation for future research in visual reasoning using transformer-based architectures.

Unraveling the ARC Puzzle: Mimicking Human Solutions with Object-Centric Decision Transformer

Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objects

Tackling the Abstraction and Reasoning Corpus (ARC) with Object-centric Models and the MDL Principle

Abductive Symbolic Solver on Abstraction and Reasoning Corpus

Abstract Reasoning with Distracting Features

Abstract Visual Reasoning Enabled by Language

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Multi-Game Decision Transformers

Predictive Coding for Decision Transformer

Combining Induction and Transduction for Abstract Reasoning

Building Human-Like Artificial Agents: A General Cognitive Algorithm for Emulating Human Decision-Making in Dynamic Environments

Solving Multi-Goal Robotic Tasks with Decision Transformer

An Approach to Solving the Abstraction and Reasoning Corpus (ARC) Challenge

Solving Reasoning Tasks with a Slot Transformer

Think Before You Act: Decision Transformers with Working Memory

Spatially-Aware Transformer for Embodied Agents

Orangutan: A Multiscale Brain Emulation-Based Artificial Intelligence Framework for Dynamic Environments

Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Training

Learning to Play Atari in a World of Tokens

Learning to Solve Abstract Reasoning Problems with Neurosymbolic Program Synthesis and Task Generation

Read to Play (R2-Play): Decision Transformer with Multimodal Game Instruction