Abstract:Responses to natural stimuli in area V4—a mid-level area of the visual ventral stream—are well predicted by features from convolutional neural networks (CNNs) trained on image classification. This result has been taken as evidence for the functional role of V4 in object classification. However, we currently do not know if and to what extent V4 plays a role in solving other computational objectives. Here, we investigated normative accounts of V4 (and V1 for comparison) by predicting macaque single-neuron responses to natural images from the representations extracted by 23 CNNs trained on different computer vision tasks including semantic, geometric, 2D, and 3D types of tasks. We found that V4 was best predicted by semantic classification features and exhibited high task selectivity, while the choice of task was less consequential to V1 performance. Consistent with traditional characterizations of V4 function that show its high-dimensional tuning to various 2D and 3D stimulus directions, we found that diverse non-semantic tasks explained aspects of V4 function that are not captured by individual semantic tasks. Nevertheless, jointly considering the features of a pair of semantic classification tasks was sufficient to yield one of our top V4 models, solidifying V4's main functional role in semantic processing and suggesting that V4's selectivity to 2D or 3D stimulus properties found by electrophysiologists can result from semantic functional goals. The functional role of area V4 in the primate visual cortex has been traditionally studied by measuring tuning properties to simple, parametric stimuli, potentially overlooking other important aspects that could be revealed with richer, natural stimuli. Here, we combine single-cell recordings of macaque V1 and V4 responses to natural images, and deep learning models trained on multiple computer vision tasks. We found that V4 responses are best predicted by representations that are critical to solve semantic tasks like object and scene classification. Moreover, our results suggest that that V4's affinity to different 2D and 3D stimulus properties likely stems from its involvement in semantic processing. Overall, our diverse task-driven modeling approach enriches our understanding of the functional role of visual areas in the brain.

Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers

BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity

Finding Shared Decodable Concepts and their Negations in the Brain

Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models

Identifying Shared Decodable Concepts in the Human Brain Using Image-Language Foundation Models

Natural speech reveals the semantic maps that tile human cerebral cortex

Brain Decodes Deep Nets

Spatial encoding of BOLD fMRI time series for categorizing static images across visual datasets: A pilot study on human vision

Distributed network flows generate localized category selectivity in human visual cortex

A large-scale examination of inductive biases shaping high-level visual representation in brains and machines

Parallel Backpropagation for Shared-Feature Visualization

Exploring the Relationship Between Visual Information and Language Semantic Concept in the Human Brain

Hierarchical Bayesian Causality Network to Extract High-Level Semantic Information in Visual Cortex

Saliency Suppressed, Semantics Surfaced: Visual Transformations in Neural Networks and the Brain

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

Diverse task-driven modeling of macaque V4 reveals functional specialization towards semantic tasks

Convergence and Divergence in the Neural Organization of Object Responses to Pictures and Words

Modeling Category-Selective Cortical Regions with Topographic Variational Autoencoders

Integrated deep visual and semantic attractor neural networks predict fMRI pattern-information along the ventral object processing pathway