Abstract:Over the past decade, predictive modeling of neural responses in the primate visual system has advanced significantly, largely driven by various DNN approaches. These include models optimized directly for visual recognition, cross-modal alignment through contrastive objectives, neural response prediction from scratch, and large language model <a class="link-external link-http" href="http://embeddings.Likewise" rel="external noopener nofollow">this http URL</a>, different readout mechanisms, ranging from fully linear to spatial-feature factorized methods have been explored for mapping network activations to neural responses. Despite the diversity of these approaches, it remains unclear which method performs best across different visual regions. In this study, we systematically compare these approaches for modeling the human visual system and investigate alternative strategies to improve response predictions. Our findings reveal that for early to mid-level visual areas, response-optimized models with visual inputs offer superior prediction accuracy, while for higher visual regions, embeddings from LLMs based on detailed contextual descriptions of images and task-optimized models pretrained on large vision datasets provide the best fit. Through comparative analysis of these modeling approaches, we identified three distinct regions in the visual cortex: one sensitive primarily to perceptual features of the input that are not captured by linguistic descriptions, another attuned to fine-grained visual details representing semantic information, and a third responsive to abstract, global meanings aligned with linguistic content. We also highlight the critical role of readout mechanisms, proposing a novel scheme that modulates receptive fields and feature maps based on semantic content, resulting in an accuracy boost of 3-23% over existing SOTAs for all models and brain regions. Together, these findings offer key insights into building more precise models of the visual system.

Representational constraints underlying similarity between task-optimized neural systems

Evaluating (and Improving) the Correspondence Between Deep Neural Networks and Human Representations

Universal dimensions of visual representation

Relational Constraints On Neural Networks Reproduce Human Biases towards Abstract Geometric Regularity

Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation

Using drawings and deep neural networks to characterize the building blocks of human visual similarity

Dimensions underlying the representational alignment of deep neural networks with humans

Evaluating Representational Similarity Measures from the Lens of Functional Correspondence

Comparison Against Task Driven Artificial Neural Networks Reveals Functional Organization of Mouse Visual Cortex

A large-scale examination of inductive biases shaping high-level visual representation in brains and machines

Human-Like Geometric Abstraction in Large Pre-trained Neural Networks

What comparing deep neural networks can teach us about human vision

Modeling the Human Visual System: Comparative Insights from Response-Optimized and Task-Optimized Vision Models, Language Models, and different Readout Mechanisms

Humans and deep networks largely agree on which kinds of variation make object recognition harder

Human perception in computer vision

Manipulating and measuring variation in deep neural network (DNN) representations of objects

To What Extent Do Different Neural Networks Learn the Same Representation: A Neuron Activation Subspace Match Approach

Representations and generalization in artificial and brain neural networks

A Hierarchical Model for Invariance of Visual Perception Based on the Biological Structure of Interotemporal Cortex

Aligning Machine and Human Visual Representations across Abstraction Levels

Exploring the Categorical Nature of Colour Perception: Insights from Artificial Networks