Abstract:Convolutional neural networks (CNNs) are one of the most successful computer vision systems to solve object recognition. Furthermore, CNNs have major applications in understanding the nature of visual representations in the human brain. Yet it remains poorly understood how CNNs actually make their decisions, what the nature of their internal representations is, and how their recognition strategies differ from humans. Specifically, there is a major debate about the question of whether CNNs primarily rely on surface regularities of objects, or whether they are capable of exploiting the spatial arrangement of features, similar to humans. Here, we develop a novel feature-scrambling approach to explicitly test whether CNNs use the spatial arrangement of features (i.e. object parts) to classify objects. We combine this approach with a systematic manipulation of effective receptive field sizes of CNNs as well as minimal recognizable configurations (MIRCs) analysis. In contrast to much previous literature, we provide evidence that CNNs are in fact capable of using relatively long-range spatial relationships for object classification. Moreover, the extent to which CNNs use spatial relationships depends heavily on the dataset, e.g. texture vs. sketch. In fact, CNNs even use different strategies for different classes within heterogeneous datasets (ImageNet), suggesting CNNs have a continuous spectrum of classification strategies. Finally, we show that CNNs learn the spatial arrangement of features only up to an intermediate level of granularity, which suggests that intermediate rather than global shape features provide the optimal trade-off between sensitivity and specificity in object classification. These results provide novel insights into the nature of CNN representations and the extent to which they rely on the spatial arrangement of features for object classification.

On The Potential of The Fractal Geometry and The CNNs Ability to Encode it

Two-Stream Boundary-Aware Neural Network for Concrete Crack Segmentation and Quantification

Fractal Dimension Invariant Filtering and Its CNN-based Implementation

Encoding Spatial Distribution of Convolutional Features for Texture Representation.

Towards Better Analysis of Deep Convolutional Neural Networks

A Scale-Invariant Diagnostic Approach Towards Understanding Dynamics of Deep Neural Networks

Deep Fractal Residual Network for Fast and Accurate Single Image Super Resolution

Modulation Signal Recognition Based on Feature Fusion and Residual CNN in Fractal Noise Background

Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks

Uncovering Nested Data Parallelism and Data Reuse in DNN Computation with FractalTensor

CNN-G: Convolutional Neural Network Combined With Graph for Image Segmentation With Theoretical Analysis

Representing Topological Self-Similarity Using Fractal Feature Maps for Accurate Segmentation of Tubular Structures

Improving Fractal Pre-training

Prediction of Fractal Dimension in Shale CT and Its Robustness to Interference Based on Convolutional Neural Networks

Refining Architectures of Deep Convolutional Neural Networks

An ensemble of learned features and reshaping of fractal geometry-based descriptors for classification of histological images

Recognition Of 3D Surface Fractal Dimension Based On Convolutional Neural Network

Deep Continuous Networks

A novel feature-scrambling approach reveals the capacity of convolutional neural networks to learn spatial relations

Fast hybrid fractal image compression using an image feature and neural network

Understanding deep features with computer-generated imagery