Modeling the Human Visual System: Comparative Insights from Response-Optimized and Task-Optimized Vision Models, Language Models, and different Readout Mechanisms

Shreya Saha,Ishaan Chadha,Meenakshi khosla

2024-10-18

Abstract:Over the past decade, predictive modeling of neural responses in the primate visual system has advanced significantly, largely driven by various DNN approaches. These include models optimized directly for visual recognition, cross-modal alignment through contrastive objectives, neural response prediction from scratch, and large language model <a class="link-external link-http" href="http://embeddings.Likewise" rel="external noopener nofollow">this http URL</a>, different readout mechanisms, ranging from fully linear to spatial-feature factorized methods have been explored for mapping network activations to neural responses. Despite the diversity of these approaches, it remains unclear which method performs best across different visual regions. In this study, we systematically compare these approaches for modeling the human visual system and investigate alternative strategies to improve response predictions. Our findings reveal that for early to mid-level visual areas, response-optimized models with visual inputs offer superior prediction accuracy, while for higher visual regions, embeddings from LLMs based on detailed contextual descriptions of images and task-optimized models pretrained on large vision datasets provide the best fit. Through comparative analysis of these modeling approaches, we identified three distinct regions in the visual cortex: one sensitive primarily to perceptual features of the input that are not captured by linguistic descriptions, another attuned to fine-grained visual details representing semantic information, and a third responsive to abstract, global meanings aligned with linguistic content. We also highlight the critical role of readout mechanisms, proposing a novel scheme that modulates receptive fields and feature maps based on semantic content, resulting in an accuracy boost of 3-23% over existing SOTAs for all models and brain regions. Together, these findings offer key insights into building more precise models of the visual system.

Neural and Evolutionary Computing,Machine Learning

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address how to construct accurate predictive models of the visual system, particularly the optimal methods for modeling neural responses in different regions of the human visual system. Specifically, the paper seeks to answer the following key questions:

1. **Performance Comparison of Different Modeling Methods**:
- In different visual cortex regions, which modeling methods (including task-optimized models, response-optimized models, language models, and different readout mechanisms) can provide the most accurate neural response predictions?
- How do these methods perform differently in early visual areas (such as V1, V2, V3, V4) and higher visual areas (such as ventral, dorsal, and lateral streams)?

2. **Response Characteristics of Different Visual Cortex Regions to Inputs**:
- By comparing the performance of different models, identify three main regions in the visual cortex that are sensitive to perceptual features, local visual semantics, and global semantic information, respectively.

3. **Improvement of Readout Mechanisms**:
- Propose a new readout mechanism (semantic readout based on spatial transformers) to improve the accuracy of neural response predictions.
- Evaluate the performance of different readout mechanisms (linear readout, Gaussian 2D readout, spatial feature decomposition linear readout, and semantic spatial transformer readout) in different visual cortex regions.

4. **Combination of Visual and Language Models**:
- Explore the role of language models (such as embeddings from large language models) in predicting neural responses in higher visual areas, especially when combined with visual inputs.

By answering these questions, the paper aims to provide key insights for constructing more accurate visual system models and propose methods to improve existing models.