Abstract:The visual system is hierarchically organized to process visual information in successive stages. Neural representations vary drastically across the first stages of visual processing: at the output of the retina, ganglion cell receptive fields (RFs) exhibit a clear antagonistic center-surround structure, whereas in the primary visual cortex, typical RFs are sharply tuned to a precise orientation. There is currently no unified theory explaining these differences in representations across layers. Here, using a deep convolutional neural network trained on image recognition as a model of the visual system, we show that such differences in representation can emerge as a direct consequence of different neural resource constraints on the retinal and cortical networks, and we find a single model from which both geometries spontaneously emerge at the appropriate stages of visual processing. The key constraint is a reduced number of neurons at the retinal output, consistent with the anatomy of the optic nerve as a stringent bottleneck. Second, we find that, for simple cortical networks, visual representations at the retinal output emerge as nonlinear and lossy feature detectors, whereas they emerge as linear and faithful encoders of the visual scene for more complex cortices. This result predicts that the retinas of small vertebrates should perform sophisticated nonlinear computations, extracting features directly relevant to behavior, whereas retinas of large animals such as primates should mostly encode the visual scene linearly and respond to a much broader range of stimuli. These predictions could reconcile the two seemingly incompatible views of the retina as either performing feature extraction or efficient coding of natural scenes, by suggesting that all vertebrates lie on a spectrum between these two objectives, depending on the degree of neural resources allocated to their visual system.

Unified theory for joint covariance properties under geometric image transformations for spatio-temporal receptive fields according to the generalized Gaussian derivative model for visual receptive fields

Covariance properties under natural image transformations for the generalized Gaussian derivative model for visual receptive fields

Relationships between the degrees of freedom in the affine Gaussian derivative model for visual receptive fields and 2-D affine image transformations, with application to covariance properties of simple cells in the primary visual cortex

Orientation selectivity properties for the affine Gaussian derivative and the affine Gabor models for visual receptive fields

Covariant spatio-temporal receptive fields for neuromorphic computing

A Spatial Concordance Correlation Coefficient with an Application to Image Analysis

The Gabor-Einstein Wavelet: A Model for the Receptive Fields of V1 to MT Neurons

Spatio-Chromatic Information available from different Neural Layers via Gaussianization

Derivatives and Inverse of Cascaded Linear+Nonlinear Neural Models

Derivatives and inverse of a linear-nonlinear multi-layer spatial vision model

Idealized computational models for auditory receptive fields

Do the receptive fields in the primary visual cortex span a variability over the degree of elongation of the receptive fields?

Geometric Analysis of the Conformal Camera for Intermediate-Level Vision and Perisaccadic Perception

Perception Of Transformation Invariance In The Visual Pathway

Visibility graph-based covariance functions for scalable spatial analysis in non-convex partially Euclidean domains

Mechanisms of Adaptive Spatial Integration in a Neural Model of Cortical Motion Processing

Non-separable Covariance Kernels for Spatiotemporal Gaussian Processes based on a Hybrid Spectral Method and the Harmonic Oscillator

A Unified Theory of Early Visual Representations from Retina to Cortex through Anatomically Constrained Deep CNNs

Hybrid Parametric Classes of Isotropic Covariance Functions for Spatial Random Fields

Improved contour detection model with spatial summation properties based on nonclassical receptive field

Adaptive Scales of Spatial Integration and Response Latencies in a Critically-Balanced Model of the Primary Visual Cortex