Abstract:Understanding the computational mechanisms that underlie the encoding and decoding of environmental stimuli is a paramount investigation within the domain of neuroscience. Central to this pursuit is the exploration of how the brain represents visual information across its hierarchical architecture. A prominent challenge resides in discerning the neural underpinnings of the processing of dynamic natural visual scenes. Although considerable research efforts have been made to characterize individual components of the visual pathway, a systematic understanding of the distinctive neural coding associated with visual stimuli, as they traverse this hierarchical landscape, remains elusive. In this study, we leverage the comprehensive Allen Visual Coding dataset and utilize the capabilities of deep learning neural network models to study the question of neural coding in response to dynamic natural visual scenes across an expansive array of brain regions. We find that our decoding model adeptly deciphers visual scenes from neural spiking patterns exhibited within each distinct brain area. A compelling observation arises from the comparative analysis of decoding performances, which manifests as a notable encoding proficiency within both the visual cortex and subcortical nuclei, in contrast to a relatively diminished encoding activity within hippocampal neurons. Strikingly, our results reveal a robust correlation between our decoding metrics and well-established anatomical and functional hierarchy indexes. These findings not only corroborate existing knowledge in visual coding using artificial visual stimuli but illuminate the functional role of these deeper brain regions using dynamic natural scenes. Consequently, our results proffer a novel perspective on the utility of decoding neural network models as a metric for quantifying the encoding of dynamic natural visual scenes, thereby advancing our comprehension of visual coding within the complex hierarchy of the brain.

DPVis: Automatic Visual Encoding Based on Deep Learning

Visual relationship detection with a deep convolutional relationship network

VisCode: Embedding Information in Visualization Images using Encoder-Decoder Network

A visual encoding model based on deep neural networks and transfer learning

GaborNet Visual Encoding: A Lightweight Region-Based Visual Encoding Model With Good Expressiveness and Biological Interpretability

Visualization Assessment - A Machine Learning Approach.

A Visual Encoding Model Based on Deep Neural Networks and Transfer Learning for Brain Activity Measured by Functional Magnetic Resonance Imaging

DeepEye: Towards Automatic Data Visualization

VisuaLizations As Intermediate Representations (VLAIR): an Approach for Applying Deep Learning-Based Computer Vision to Non-Image-based Data

NeuralVis: Visualizing and Interpreting Deep Learning Models

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision

ViVA: Semi-Supervised Visualization Via Variational Autoencoders

Deep Learning for Visual Data Compression

DEEPEYE: an Automatic Big Data Visualization Framework

Deep Learning based Visually Rich Document Content Understanding: A Survey

Deep Learning-Based Video Coding

DVPE: Divided View Position Embedding for Multi-View 3D Object Detection

Visual Point Cloud Forecasting enables Scalable Autonomous Driving

Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning “What” and “Where”

Unveiling Encoder-Free Vision-Language Models

Decoding dynamic visual scenes across the brain hierarchy