Abstract:Over the past decade, predictive modeling of neural responses in the primate visual system has advanced significantly, largely driven by various DNN approaches. These include models optimized directly for visual recognition, cross-modal alignment through contrastive objectives, neural response prediction from scratch, and large language model <a class="link-external link-http" href="http://embeddings.Likewise" rel="external noopener nofollow">this http URL</a>, different readout mechanisms, ranging from fully linear to spatial-feature factorized methods have been explored for mapping network activations to neural responses. Despite the diversity of these approaches, it remains unclear which method performs best across different visual regions. In this study, we systematically compare these approaches for modeling the human visual system and investigate alternative strategies to improve response predictions. Our findings reveal that for early to mid-level visual areas, response-optimized models with visual inputs offer superior prediction accuracy, while for higher visual regions, embeddings from LLMs based on detailed contextual descriptions of images and task-optimized models pretrained on large vision datasets provide the best fit. Through comparative analysis of these modeling approaches, we identified three distinct regions in the visual cortex: one sensitive primarily to perceptual features of the input that are not captured by linguistic descriptions, another attuned to fine-grained visual details representing semantic information, and a third responsive to abstract, global meanings aligned with linguistic content. We also highlight the critical role of readout mechanisms, proposing a novel scheme that modulates receptive fields and feature maps based on semantic content, resulting in an accuracy boost of 3-23% over existing SOTAs for all models and brain regions. Together, these findings offer key insights into building more precise models of the visual system.

Towards Human-Leveled Vision Systems

VBNet: A Visually-Aware Biomimetic Network for Simulating the Human Eye's Visual System

Inferring Human Vision in a Human-Like Way: Key Factors Influencing the Cognitive Processing of Level-1 Visual Perspective-Taking

Deep neural networks: a new framework for modelling biological vision and brain information processing

Review of development of visual neural computing

Hierarchies in Visual Pathway: Functions and Inspired Artificial Vision

Nearing or Surpassing: Overall Evaluation of Human-Machine Dynamic Vision Ability

Image Feature Extraction and Object Recognition Based on Vision Neural Mechanism

Human perception in computer vision

Deep Learning for Visual Neuroprosthesis

Modeling the Human Visual System: Comparative Insights from Response-Optimized and Task-Optimized Vision Models, Language Models, and different Readout Mechanisms

Aligning Machine and Human Visual Representations across Abstraction Levels

Visualizing and Understanding Vision System

Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

The Simulation of Early Vision in Biological Retina and Analysis of Its Performance

Research on Neural Encoding Models for Biological Vision: Progress and Challenges

A retina-inspired neurocomputing circuit for image representation

Modelling Human Visual Motion Processing with Trainable Motion Energy Sensing and a Self-attention Network

Human Centric Visual Analysis with Deep Learning

Convolutional neural networks for vision neuroscience: significance, developments, and outstanding issues

Multifunctional human visual pathway-replicated hardware based on 2D materials