Abstract:A fundamental component of human vision is our ability to parse complex visual scenes and judge the relations between their constituent objects. AI benchmarks for visual reasoning have driven rapid progress in recent years with state-of-the-art systems now reaching human accuracy on some of these benchmarks. Yet, a major gap remains in terms of the sample efficiency with which humans and AI systems learn new visual reasoning tasks. Humans' remarkable efficiency at learning has been at least partially attributed to their ability to harness compositionality -- such that they can efficiently take advantage of previously gained knowledge when learning new tasks. Here, we introduce a novel visual reasoning benchmark, Compositional Visual Relations (CVR), to drive progress towards the development of more data-efficient learning algorithms. We take inspiration from fluidic intelligence and non-verbal reasoning tests and describe a novel method for creating compositions of abstract rules and associated image datasets at scale. Our proposed benchmark includes measures of sample efficiency, generalization and transfer across task rules, as well as the ability to leverage compositionality. We systematically evaluate modern neural architectures and find that, surprisingly, convolutional architectures surpass transformer-based architectures across all performance measures in most data regimes. However, all computational models are a lot less data efficient compared to humans even after learning informative visual representations using self-supervision. Overall, we hope that our challenge will spur interest in the development of neural architectures that can learn to harness compositionality toward more efficient learning.

V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices

A Benchmark for Compositional Visual Reasoning

PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning

Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding

Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions

Learning Visual Reasoning Without Strong Priors

A Cognitively-Inspired Neural Architecture for Visual Abstract Reasoning Using Contrastive Perceptual and Conceptual Processing

Understanding the computational demands underlying visual reasoning

What is the Visual Cognition Gap between Humans and Multimodal LLMs?

Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representation

ViTCN: Vision Transformer Contrastive Network For Reasoning

MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning

RAVEN: A Dataset for Relational and Analogical Visual rEasoNing

Take A Step Back: Rethinking the Two Stages in Visual Reasoning

A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories

Multi-Granularity Modularized Network for Abstract Visual Reasoning

Learning to reason over visual objects

Beyond Visual Understanding: Introducing PARROT-360V for Vision Language Model Benchmarking

Benchmarking Sequential Visual Input Reasoning and Prediction in Multimodal Large Language Models

A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models