Abstract:Understanding the computational mechanisms that underlie the encoding and decoding of environmental stimuli is a paramount investigation within the domain of neuroscience. Central to this pursuit is the exploration of how the brain represents visual information across its hierarchical architecture. A prominent challenge resides in discerning the neural underpinnings of the processing of dynamic natural visual scenes. Although considerable research efforts have been made to characterize individual components of the visual pathway, a systematic understanding of the distinctive neural coding associated with visual stimuli, as they traverse this hierarchical landscape, remains elusive. In this study, we leverage the comprehensive Allen Visual Coding dataset and utilize the capabilities of deep learning neural network models to study the question of neural coding in response to dynamic natural visual scenes across an expansive array of brain regions. We find that our decoding model adeptly deciphers visual scenes from neural spiking patterns exhibited within each distinct brain area. A compelling observation arises from the comparative analysis of decoding performances, which manifests as a notable encoding proficiency within both the visual cortex and subcortical nuclei, in contrast to a relatively diminished encoding activity within hippocampal neurons. Strikingly, our results reveal a robust correlation between our decoding metrics and well-established anatomical and functional hierarchy indexes. These findings not only corroborate existing knowledge in visual coding using artificial visual stimuli but illuminate the functional role of these deeper brain regions using dynamic natural scenes. Consequently, our results proffer a novel perspective on the utility of decoding neural network models as a metric for quantifying the encoding of dynamic natural visual scenes, thereby advancing our comprehension of visual coding within the complex hierarchy of the brain.

Exploring the Brain-like Properties of Deep Neural Networks: A Neural Encoding Perspective

Enhancing neural encoding models for naturalistic perception with a multi-level integration of deep neural networks and cortical networks

Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning “What” and “Where”

Brain Decodes Deep Nets

Visio-Linguistic Brain Encoding

Brain-like emergent properties in deep networks: impact of network architecture, datasets and training

Explaining V1 Properties with a Biologically Constrained Deep Learning Architecture

Deep neural networks: a new framework for modelling biological vision and brain information processing

Convolutional neural networks for vision neuroscience: significance, developments, and outstanding issues

Visual Encoding and Decoding of the Human Brain Based on Shared Features

Research on Neural Encoding Models for Biological Vision: Progress and Challenges

Aligning Neuronal Coding of Dynamic Visual Scenes with Foundation Vision Models

VTVBrain: A Two-stage Brain Encoding Model for Decoding Key Neural Responses in Multimodal Contexts

Biologically Realistic Computational Primitives of Neocortex Implemented on Neuromorphic Hardware Improve Vision Transformer Performance

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision

GaborNet Visual Encoding: A Lightweight Region-Based Visual Encoding Model With Good Expressiveness and Biological Interpretability

Dynamics Based Neural Encoding with Inter-Intra Region Connectivity

A Rich Source of Labels for Deep Network Models of the Primate Dorsal Visual Stream

Brain-inspired Models for Visual Object Recognition: an Overview

Upgrading Voxel-wise Encoding Model via Integrated Integration over Features and Brain Networks

Decoding dynamic visual scenes across the brain hierarchy