Abstract:Geometric descriptions of deep neural networks (DNNs) have the potential to uncover core representational principles of computational models in neuroscience. Here we examined the geometry of DNN models of visual cortex by quantifying the latent dimensionality of their natural image representations. A popular view holds that optimal DNNs compress their representations onto low-dimensional subspaces to achieve invariance and robustness, which suggests that better models of visual cortex should have lower dimensional geometries. Surprisingly, we found a strong trend in the opposite direction—neural networks with high-dimensional image subspaces tended to have better generalization performance when predicting cortical responses to held-out stimuli in both monkey electrophysiology and human fMRI data. Moreover, we found that high dimensionality was associated with better performance when learning new categories of stimuli, suggesting that higher dimensional representations are better suited to generalize beyond their training domains. These findings suggest a general principle whereby high-dimensional geometry confers computational benefits to DNN models of visual cortex. The effective dimensionality of neural population codes in both brains and artificial neural networks can be far smaller than the number of neurons in the population. In vision, it has been argued that there are crucial benefits of representing images using codes that are as simple and low-dimensional as possible, allowing representations to emphasize key semantic content and attenuate irrelevant perceptual details. However, there are competing benefits of high-dimensional codes, which can capture rich perceptual information to better support an open-ended set of visual behaviors. We quantified the effective dimensionality of neural networks created using a diverse set of training paradigms and compared these networks with image-evoked codes in the visual cortex of both monkeys and humans. Our findings revealed striking benefits of networks with higher effective dimensionality, which were consistently better predictors of cortical activity patterns and were able to readily learn new categories of images outside of their training domains. Together, these findings demonstrate the computational benefits of high-dimensional sensory codes, and they suggest that current neural network models of visual cortex may be better understood in terms of the richness of their representations rather than the details of their training tasks.

Universal scale-free representations in human visual cortex

Universal dimensions of visual representation

Distributed representations of behavior-derived object dimensions in the human visual system

Distributed representations of behaviour-derived object dimensions in the human visual system

Invariant inter-subject relational structures in the human visual cortex

Revisiting the high-dimensional geometry of population responses in visual cortex

A Unified Theory of Early Visual Representations from Retina to Cortex through Anatomically Constrained Deep CNNs

Functional organization and natural scene responses across mouse visual cortical areas revealed with encoding manifolds

Neural Coding of Global Form in the Human Visual Cortex.

Multivariate Analysis of BOLD Activation Patterns Recovers Graded Depth Representations in Human Visual and Parietal Cortex

Population encoding of stimulus features along the visual hierarchy

Inferring Population Responses in Human Visual Cortex with Classification Analysis

A large-scale, standardized physiological survey reveals higher order coding throughout the mouse visual cortex

Multi-task representations in human cortex transform along a sensory-to-motor hierarchy

High-performing neural network models of visual cortex benefit from high latent dimensionality

Neural Coding of Global Form in the Human Visual

Large-scale calcium imaging reveals a systematic V4 map for encoding natural scenes

Cortical networks of dynamic scene category representation in the human brain

Rarely categorical, always high-dimensional: how the neural code changes along the cortical hierarchy

Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition

Multi-scale spiking network model of human cerebral cortex