Abstract:Convolutional neural networks (CNNs) are one of the most successful computer vision systems to solve object recognition. Furthermore, CNNs have major applications in understanding the nature of visual representations in the human brain. Yet it remains poorly understood how CNNs actually make their decisions, what the nature of their internal representations is, and how their recognition strategies differ from humans. Specifically, there is a major debate about the question of whether CNNs primarily rely on surface regularities of objects, or whether they are capable of exploiting the spatial arrangement of features, similar to humans. Here, we develop a novel feature-scrambling approach to explicitly test whether CNNs use the spatial arrangement of features (i.e. object parts) to classify objects. We combine this approach with a systematic manipulation of effective receptive field sizes of CNNs as well as minimal recognizable configurations (MIRCs) analysis. In contrast to much previous literature, we provide evidence that CNNs are in fact capable of using relatively long-range spatial relationships for object classification. Moreover, the extent to which CNNs use spatial relationships depends heavily on the dataset, e.g. texture vs. sketch. In fact, CNNs even use different strategies for different classes within heterogeneous datasets (ImageNet), suggesting CNNs have a continuous spectrum of classification strategies. Finally, we show that CNNs learn the spatial arrangement of features only up to an intermediate level of granularity, which suggests that intermediate rather than global shape features provide the optimal trade-off between sensitivity and specificity in object classification. These results provide novel insights into the nature of CNN representations and the extent to which they rely on the spatial arrangement of features for object classification.

Structured Receptive Fields in CNNs

Untrained Neural Network Enabling Fast and Universal Structured-Illumination Microscopy

SUBP: Soft Uniform Block Pruning for 1 X N Sparse CNNs Multithreading Acceleration

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

SUBP: Soft Uniform Block Pruning for 1xn Sparse CNNs Multithreading Acceleration

Revealing Fine Structures of the Retinal Receptive Field by Deep-Learning Networks

Adversarial Structured Neural Network Pruning

Remote Sensing Scene Classification Using Sparse Representation-Based Framework with Deep Feature Fusion

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Initialization of CNN Models for Training on a Small Dataset Using Importance of Filter Parameters

Adaptive Structured Sparse Network for Efficient CNNs with Feature Regularization.

Learning Structure and Strength of CNN Filters for Small Sample Size Training

Sparse convolutional neural network for high-resolution skull shape completion and shape super-resolution

Learning Structured Sparsity in Deep Neural Networks

Adaptive Structured Sparse Network for Efficient CNNs with Feature Regularization

Scattering Networks for Hybrid Representation Learning

Towards Optimal Structured Cnn Pruning Via Generative Adversarial Learning

A novel feature-scrambling approach reveals the capacity of convolutional neural networks to learn spatial relations

Learning From Brains How to Regularize Machines