Abstract:Our brains represent the ever-changing environment with neurons in a highly dynamic fashion. The temporal features of visual pixels in dynamic natural scenes are entrapped in the neuronal responses of the retina. It is crucial to establish the intrinsic temporal relationship between visual pixels and neuronal responses. Recent foundation vision models have paved an advanced way of understanding image pixels. Yet, neuronal coding in the brain largely lacks a deep understanding of its alignment with pixels. Most previous studies employ static images or artificial videos derived from static images for emulating more real and complicated stimuli. Despite these simple scenarios effectively help to separate key factors influencing visual coding, complex temporal relationships receive no consideration. To decompose the temporal features of visual coding in natural scenes, here we propose Vi-ST, a spatiotemporal convolutional neural network fed with a self-supervised Vision Transformer (ViT) prior, aimed at unraveling the temporal-based encoding patterns of retinal neuronal populations. The model demonstrates robust predictive performance in generalization tests. Furthermore, through detailed ablation experiments, we demonstrate the significance of each temporal module. Furthermore, we introduce a visual coding evaluation metric designed to integrate temporal considerations and compare the impact of different numbers of neuronal populations on complementary coding. In conclusion, our proposed Vi-ST demonstrates a novel modeling framework for neuronal coding of dynamic visual scenes in the brain, effectively aligning our brain representation of video with neuronal activity. The code is available at <a class="link-external link-https" href="https://github.com/wurining/Vi-ST" rel="external noopener nofollow">this https URL</a>.

Spatio-Temporal Adaptation in the Unsupervised Development of Networked Visual Neurons

Vision attention learning model and its application in robot

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

A Spatial–Channel–Temporal-Fused Attention for Spiking Neural Networks

Spatially-Aware Context Neural Networks.

Adaptive spatiotemporal neural networks through complementary hybridization

Advancing Spatio-Temporal Processing in Spiking Neural Networks through Adaptation

Exploiting Neuron and Synapse Filter Dynamics in Spatial Temporal Learning of Deep Spiking Neural Network

STSC-SNN: Spatio-Temporal Synaptic Connection with temporal convolution and attention for spiking neural networks

Temporal dynamics of short-term neural adaptation across human visual cortex

Aligning Neuronal Coding of Dynamic Visual Scenes with Foundation Vision Models

Spatial-Temporal Self-Attention for Asynchronous Spiking Neural Networks

Spatial Scene Representations Formed by Self‐organizing Learning in a Hippocampal Extension of the Ventral Visual System

Temporal Conditioning Spiking Latent Variable Models of the Neural Response to Natural Visual Scenes

Adaptation shapes the representational geometry in mouse V1 to efficiently encode the environment

Enhancing SNN-based Spatio-Temporal Learning: A Benchmark Dataset and Cross-Modality Attention Model

Modelling Human Visual Motion Processing with Trainable Motion Energy Sensing and a Self-attention Network

Autaptic Synaptic Circuit Enhances Spatio-temporal Predictive Learning of Spiking Neural Networks

Attention Embedded Spatio-Temporal Network for Video Salient Object Detection

Visual Attention Modeling Based on Short-Term Environmental Adaption