Abstract:Understanding the computational mechanisms that underlie the encoding and decoding of environmental stimuli is a paramount investigation within the domain of neuroscience. Central to this pursuit is the exploration of how the brain represents visual information across its hierarchical architecture. A prominent challenge resides in discerning the neural underpinnings of the processing of dynamic natural visual scenes. Although considerable research efforts have been made to characterize individual components of the visual pathway, a systematic understanding of the distinctive neural coding associated with visual stimuli, as they traverse this hierarchical landscape, remains elusive. In this study, we leverage the comprehensive Allen Visual Coding dataset and utilize the capabilities of deep learning neural network models to study the question of neural coding in response to dynamic natural visual scenes across an expansive array of brain regions. We find that our decoding model adeptly deciphers visual scenes from neural spiking patterns exhibited within each distinct brain area. A compelling observation arises from the comparative analysis of decoding performances, which manifests as a notable encoding proficiency within both the visual cortex and subcortical nuclei, in contrast to a relatively diminished encoding activity within hippocampal neurons. Strikingly, our results reveal a robust correlation between our decoding metrics and well-established anatomical and functional hierarchy indexes. These findings not only corroborate existing knowledge in visual coding using artificial visual stimuli but illuminate the functional role of these deeper brain regions using dynamic natural scenes. Consequently, our results proffer a novel perspective on the utility of decoding neural network models as a metric for quantifying the encoding of dynamic natural visual scenes, thereby advancing our comprehension of visual coding within the complex hierarchy of the brain.

Effective Ensemble of Deep Neural Networks Predicts Neural Responses to Naturalistic Videos

Enhancing neural encoding models for naturalistic perception with a multi-level integration of deep neural networks and cortical networks

The Algonauts Project 2023 Challenge: UARK-UAlbany Team Solution

Dynamics Based Neural Encoding with Inter-Intra Region Connectivity

Exploration and Comparison of Deep Learning Architectures to Predict Brain Response to Realistic Pictures

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision

Predicting Single Neuron Responses of the Primary Visual Cortex with Deep Learning Model

The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes

A visual encoding model based on deep neural networks and transfer learning

Diversity encouraged learning of unsupervised LSTM ensemble for neural activity video prediction

Relating CNNs with brain: Challenges and findings

Probing neural representations of scene perception in a hippocampally dependent task using artificial neural networks

One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment

Generalizability analysis of deep learning predictions of human brain responses to augmented and semantically novel visual stimuli

Decoding Neural Responses in Mouse Visual Cortex through a Deep Neural Network

Decoding dynamic visual scenes across the brain hierarchy

Exploring the Brain-like Properties of Deep Neural Networks: A Neural Encoding Perspective

Retrospective for the Dynamic Sensorium Competition for predicting large-scale mouse primary visual cortex activity from videos

Complex Properties of Training Stimuli Affect Brain Alignment in a Deep Network Model of Mouse Visual Cortex

A shared neural encoding model for the prediction of subject-specific fMRI response

Structurally-constrained encoding framework using a multi-voxel reduced-rank latent model for human natural vision