Abstract:The visual system is hierarchically organized to process visual information in successive stages. Neural representations vary drastically across the first stages of visual processing: at the output of the retina, ganglion cell receptive fields (RFs) exhibit a clear antagonistic center-surround structure, whereas in the primary visual cortex, typical RFs are sharply tuned to a precise orientation. There is currently no unified theory explaining these differences in representations across layers. Here, using a deep convolutional neural network trained on image recognition as a model of the visual system, we show that such differences in representation can emerge as a direct consequence of different neural resource constraints on the retinal and cortical networks, and we find a single model from which both geometries spontaneously emerge at the appropriate stages of visual processing. The key constraint is a reduced number of neurons at the retinal output, consistent with the anatomy of the optic nerve as a stringent bottleneck. Second, we find that, for simple cortical networks, visual representations at the retinal output emerge as nonlinear and lossy feature detectors, whereas they emerge as linear and faithful encoders of the visual scene for more complex cortices. This result predicts that the retinas of small vertebrates should perform sophisticated nonlinear computations, extracting features directly relevant to behavior, whereas retinas of large animals such as primates should mostly encode the visual scene linearly and respond to a much broader range of stimuli. These predictions could reconcile the two seemingly incompatible views of the retina as either performing feature extraction or efficient coding of natural scenes, by suggesting that all vertebrates lie on a spectrum between these two objectives, depending on the degree of neural resources allocated to their visual system.

Brain-like representational straightening of natural movies in robust feedforward neural networks

Learning From Brains How to Regularize Machines

Long-Range Feedback Spiking Network Captures Dynamic and Static Representations of the Visual Cortex under Movie Stimuli

Brain-like Flexible Visual Inference by Harnessing Feedback-Feedforward Alignment

Learning intermediate-level representations of form and motion from natural movies

Unsupervised learning on spontaneous retinal activity leads to efficient neural representation geometry

Invariant recognition drives neural representations of action sequences

Brain inspired Robust Vision using Convolutional Neural Networks with Feedback

Neural Representations of Dynamic Visual Stimuli

Saliency Suppressed, Semantics Surfaced: Visual Transformations in Neural Networks and the Brain

Optimizing deep video representation to match brain activity

Factored Neural Representation for Scene Understanding

A feedforward mechanism for human-like contour integration

A Unified Theory of Early Visual Representations from Retina to Cortex through Anatomically Constrained Deep CNNs

Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language

A Brain-Inspired Regularizer for Adversarial Robustness

Visual Image Reconstructed Without Semantics from Human Brain Activity Using Linear Image Decoders and Nonlinear Noise Suppression

Complex Properties of Training Stimuli Affect Brain Alignment in a Deep Network Model of Mouse Visual Cortex

Dissociable Neural Representations of Adversarially Perturbed Images in Convolutional Neural Networks and the Human Brain

Leveraging the Human Ventral Visual Stream to Improve Neural Network Robustness

Relational Constraints On Neural Networks Reproduce Human Biases towards Abstract Geometric Regularity