Abstract:Convolutional neural networks (CNNs) are one of the most successful computer vision systems to solve object recognition. Furthermore, CNNs have major applications in understanding the nature of visual representations in the human brain. Yet it remains poorly understood how CNNs actually make their decisions, what the nature of their internal representations is, and how their recognition strategies differ from humans. Specifically, there is a major debate about the question of whether CNNs primarily rely on surface regularities of objects, or whether they are capable of exploiting the spatial arrangement of features, similar to humans. Here, we develop a novel feature-scrambling approach to explicitly test whether CNNs use the spatial arrangement of features (i.e. object parts) to classify objects. We combine this approach with a systematic manipulation of effective receptive field sizes of CNNs as well as minimal recognizable configurations (MIRCs) analysis. In contrast to much previous literature, we provide evidence that CNNs are in fact capable of using relatively long-range spatial relationships for object classification. Moreover, the extent to which CNNs use spatial relationships depends heavily on the dataset, e.g. texture vs. sketch. In fact, CNNs even use different strategies for different classes within heterogeneous datasets (ImageNet), suggesting CNNs have a continuous spectrum of classification strategies. Finally, we show that CNNs learn the spatial arrangement of features only up to an intermediate level of granularity, which suggests that intermediate rather than global shape features provide the optimal trade-off between sensitivity and specificity in object classification. These results provide novel insights into the nature of CNN representations and the extent to which they rely on the spatial arrangement of features for object classification.

Evaluating CNNs on the Gestalt Principle of Closure

Investigating the Gestalt Principle of Closure in Deep Convolutional Neural Networks

Finding Closure: A Closer Look at the Gestalt Law of Closure in Convolutional Neural Networks

Understanding Deep Convolutional Networks through Gestalt Theory

Towards Better Analysis of Deep Convolutional Neural Networks

Relating Input Concepts to Convolutional Neural Network Decisions

Relating CNNs with brain: Challenges and findings

A novel feature-scrambling approach reveals the capacity of convolutional neural networks to learn spatial relations

A Visual Cortex-Attentive Deep Convolutional Neural Network for Digital Image Design

Refining Architectures of Deep Convolutional Neural Networks

Do DNNs trained on Natural Images acquire Gestalt Properties?

Partial success in closing the gap between human and machine vision

Do humans and Convolutional Neural Networks attend to similar areas during scene classification: Effects of task and image type

Hierarchical Layout-Aware Graph Convolutional Network for Unified Aesthetics Assessment.

Human-Like Geometric Abstraction in Large Pre-trained Neural Networks

KerCNNs: biologically inspired lateral connections for classification of corrupted images

Dissociable Neural Representations of Adversarially Perturbed Images in Convolutional Neural Networks and the Human Brain

Visualizing and Comparing Convolutional Neural Networks

Convolutional neural network-based image representation for visual loop closure detection

From convolutional neural networks to models of higher‐level cognition (and back again)

Degraded Polygons Raise Fundamental Questions of Neural Network Perception