Abstract:Convolutional neural networks (CNNs) are one of the most successful computer vision systems to solve object recognition. Furthermore, CNNs have major applications in understanding the nature of visual representations in the human brain. Yet it remains poorly understood how CNNs actually make their decisions, what the nature of their internal representations is, and how their recognition strategies differ from humans. Specifically, there is a major debate about the question of whether CNNs primarily rely on surface regularities of objects, or whether they are capable of exploiting the spatial arrangement of features, similar to humans. Here, we develop a novel feature-scrambling approach to explicitly test whether CNNs use the spatial arrangement of features (i.e. object parts) to classify objects. We combine this approach with a systematic manipulation of effective receptive field sizes of CNNs as well as minimal recognizable configurations (MIRCs) analysis. In contrast to much previous literature, we provide evidence that CNNs are in fact capable of using relatively long-range spatial relationships for object classification. Moreover, the extent to which CNNs use spatial relationships depends heavily on the dataset, e.g. texture vs. sketch. In fact, CNNs even use different strategies for different classes within heterogeneous datasets (ImageNet), suggesting CNNs have a continuous spectrum of classification strategies. Finally, we show that CNNs learn the spatial arrangement of features only up to an intermediate level of granularity, which suggests that intermediate rather than global shape features provide the optimal trade-off between sensitivity and specificity in object classification. These results provide novel insights into the nature of CNN representations and the extent to which they rely on the spatial arrangement of features for object classification.

Category-orthogonal object features guide information processing in recurrent neural networks trained for object categorization

Task-Driven Convolutional Recurrent Models of the Visual System

Characterising representation dynamics in recurrent neural networks for object recognition

Basic Level Categorization Facilitates Visual Object Recognition

Recurrent issues with deep neural networks of visual recognition

Category Decoding of Visual Stimuli From Human Brain Activity Using a Bidirectional Recurrent Neural Network to Simulate Bidirectional Information Flows in Human Visual Cortices

Geometry of naturalistic object representations in recurrent neural network models of working memory

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network.

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Recurrent Feedback Improves Feedforward Representations in Deep Neural Networks

An Event-Driven Categorization Model for AER Image Sensors Using Multispike Encoding and Learning

Recurrent connections facilitate occluded object recognition by explaining-away

Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior

Transform-Invariant Recognition by Association in a Recurrent Network

Recurrent Feedback Improves Recognition of Partially Occluded Objects

Learning The Sequential Temporal Information with Recurrent Neural Networks

How Deep is the Feature Analysis underlying Rapid Visual Categorization?

Long-Term Recurrent Convolutional Networks for Visual Recognition and Description

Recurrent Models of Visual Attention

Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition

A novel feature-scrambling approach reveals the capacity of convolutional neural networks to learn spatial relations