Abstract:Exploiting data invariances is crucial for efficient learning in both artificial and biological neural circuits. Understanding how neural networks can discover appropriate representations capable of harnessing the underlying symmetries of their inputs is thus crucial in machine learning and neuroscience. Convolutional neural networks, for example, were designed to exploit translation symmetry and their capabilities triggered the first wave of deep learning successes. However, learning convolutions directly from translation-invariant data with a fully-connected network has so far proven elusive. Here, we show how initially fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs, resulting in localised, space-tiling receptive fields. These receptive fields match the filters of a convolutional network trained on the same task. By carefully designing data models for the visual scene, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs, which has long been recognised as the hallmark of natural images. We provide an analytical and numerical characterisation of the pattern-formation mechanism responsible for this phenomenon in a simple model and find an unexpected link between receptive field formation and tensor decomposition of higher-order input correlations. These results provide a new perspective on the development of low-level feature detectors in various sensory modalities, and pave the way for studying the impact of higher-order statistics on learning in neural networks.

Understanding the Error Structure as a Key to Regularize Convolutional Neural Networks

LoSS: Local Structural Separation Hypergraph Convolutional Neural Network

Inter-Class Angular Loss for Convolutional Neural Networks.

Hierarchical Gate Network for Fine-Grained Visual Recognition.

Learning Structures for Deep Neural Networks

Joint Structure Similarity and Class Information for Image Classification

IC-Network: Efficient Structure for Convolutional Neural Networks

Regularizing Deep Convolutional Neural Networks with a Structured Decorrelation Constraint.

IC Networks: Remodeling the Basic Unit for Convolutional Neural Networks

Learning Structured and Non-Redundant Representations with Deep Neural Networks

Learning From Brains How to Regularize Machines

Visualizing and Comparing Convolutional Neural Networks

On the Learning Dynamics of Two-layer Nonlinear Convolutional Neural Networks.

Confusion-Aware Convolutional Neural Network For Image Classification

Error-Driven Incremental Learning in Deep Convolutional Neural Network for Large-Scale Image Classification

Structured Convolutions for Efficient Neural Network Design

Visualizing and Understanding Convolutional Networks

DBNet: A New Generalized Structure Efficient for Classification

Towards Better Analysis of Deep Convolutional Neural Networks

Do Convolutional Neural Networks Learn Class Hierarchy?

Data-driven emergence of convolutional structure in neural networks