Abstract:Deep neural networks (DNNs) are promising models of the cortical computations supporting human object recognition. However, despite their ability to explain a significant portion of variance in neural data, the agreement between models and brain representational dynamics is far from perfect. We address this issue by asking which representational features are currently unaccounted for in neural time series data, estimated for multiple areas of the ventral stream via source-reconstructed magnetoencephalography data acquired in human participants (nine females, six males) during object viewing. We focus on the ability of visuo-semantic models, consisting of human-generated labels of object features and categories, to explain variance beyond the explanatory power of DNNs alone. We report a gradual reversal in the relative importance of DNN versus visuo-semantic features as ventral-stream object representations unfold over space and time. Although lower-level visual areas are better explained by DNN features starting early in time (at 66 ms after stimulus onset), higher-level cortical dynamics are best accounted for by visuo-semantic features starting later in time (at 146 ms after stimulus onset). Among the visuo-semantic features, object parts and basic categories drive the advantage over DNNs. These results show that a significant component of the variance unexplained by DNNs in higher-level cortical dynamics is structured and can be explained by readily nameable aspects of the objects. We conclude that current DNNs fail to fully capture dynamic representations in higher-level human visual cortex and suggest a path toward more accurate models of ventral-stream computations.SIGNIFICANCE STATEMENT When we view objects such as faces and cars in our visual environment, their neural representations dynamically unfold over time at a millisecond scale. These dynamics reflect the cortical computations that support fast and robust object recognition. DNNs have emerged as a promising framework for modeling these computations but cannot yet fully account for the neural dynamics. Using magnetoencephalography data acquired in human observers during object viewing, we show that readily nameable aspects of objects, such as 'eye', 'wheel', and 'face', can account for variance in the neural dynamics over and above DNNs. These findings suggest that DNNs and humans may in part rely on different object features for visual recognition and provide guidelines for model improvement.

Neural Dynamics of Object Manifold Alignment in the Ventral Stream

Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences

Deep Neural Networks and Visuo-Semantic Models Explain Complementary Components of Human Ventral-Stream Representational Dynamics

Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation

Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations

Beyond Sight: Probing Alignment Between Image Models and Blind V1

Complex Properties of Training Stimuli Affect Brain Alignment in a Deep Network Model of Mouse Visual Cortex

One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment

Leveraging the Human Ventral Visual Stream to Improve Neural Network Robustness

Evaluating alignment between humans and neural network representations in image-based learning tasks

Deep generative networks reveal the tuning of neurons in IT and predict their influence on visual perception

Achieving More Human Brain-Like Vision via Human EEG Representational Alignment

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream

Functional organization and natural scene responses across mouse visual cortical areas revealed with encoding manifolds

AlignedCut: Visual Concepts Discovery on Brain-Guided Universal Feature Space

Neural manifolds in V1 change with top-down signals from V4 targeting the foveal region

Non-accidental Properties, Metric Invariance, and Encoding by Neurons in a Model of Ventral Stream Visual Object Recognition, VisNet

Adaptive stretching of representations across brain regions and deep learning model layers

Invariant Visual Object and Face Recognition: Neural and Computational Bases, and a Model, VisNet

Identifying Interpretable Visual Features in Artificial and Biological Neural Systems