Abstract:Convolutional neural networks (CNNs) are one of the most successful computer vision systems to solve object recognition. Furthermore, CNNs have major applications in understanding the nature of visual representations in the human brain. Yet it remains poorly understood how CNNs actually make their decisions, what the nature of their internal representations is, and how their recognition strategies differ from humans. Specifically, there is a major debate about the question of whether CNNs primarily rely on surface regularities of objects, or whether they are capable of exploiting the spatial arrangement of features, similar to humans. Here, we develop a novel feature-scrambling approach to explicitly test whether CNNs use the spatial arrangement of features (i.e. object parts) to classify objects. We combine this approach with a systematic manipulation of effective receptive field sizes of CNNs as well as minimal recognizable configurations (MIRCs) analysis. In contrast to much previous literature, we provide evidence that CNNs are in fact capable of using relatively long-range spatial relationships for object classification. Moreover, the extent to which CNNs use spatial relationships depends heavily on the dataset, e.g. texture vs. sketch. In fact, CNNs even use different strategies for different classes within heterogeneous datasets (ImageNet), suggesting CNNs have a continuous spectrum of classification strategies. Finally, we show that CNNs learn the spatial arrangement of features only up to an intermediate level of granularity, which suggests that intermediate rather than global shape features provide the optimal trade-off between sensitivity and specificity in object classification. These results provide novel insights into the nature of CNN representations and the extent to which they rely on the spatial arrangement of features for object classification.

Learning and Memory of Spatial Relationship by a Neural Network with Sparse Features

Spatial Relationship Representation for Visual Object Searching

Learning Object Spatial Relationship from Demonstration

Exploit Spatiotemporal Contextual Information for 3D Single Object Tracking Via Memory Networks

Spatially-Aware Context Neural Networks.

Searching Eye Centers Using a Context-Based Neural Network

Sparse Temporal Encoding of Visual Features for Robust Object Recognition by Spiking Neurons

A Brain-Inspired Spiking Neural Network Model with Temporal Encoding and Learning

Learning to Match Features with Discriminative Sparse Graphneuralnetwork

Sparse Coding of Visual Context

A Navigation Cognitive System Driven by Hierarchical Spiking Neural Network.

Spatial Relation Learning in Complementary Scenarios with Deep Neural Networks.

A novel feature-scrambling approach reveals the capacity of convolutional neural networks to learn spatial relations

Learning Low-Rank Sparse Representations With Robust Relationship Inference for Image Memorability Prediction

A Novel Neural Multi-Store Memory Network for Autonomous Visual Navigation in Unknown Environment

Relationship Learning From Multisource Images via Spatial-Spectral Perception Network

Learning Allocentric Representations of Space for Navigation

Competitive learning to generate sparse representations for associative memory

Sparse Representation and Learning in Visual Recognition: Theory and Applications

Semantic Spatiotemporal Memory Toward 3D Robotic Vision

Learning-Disability Recognition by Using Sparse Spatio-Temporal Graph Neural Networks.