Abstract:Our brains represent the ever-changing environment with neurons in a highly dynamic fashion. The temporal features of visual pixels in dynamic natural scenes are entrapped in the neuronal responses of the retina. It is crucial to establish the intrinsic temporal relationship between visual pixels and neuronal responses. Recent foundation vision models have paved an advanced way of understanding image pixels. Yet, neuronal coding in the brain largely lacks a deep understanding of its alignment with pixels. Most previous studies employ static images or artificial videos derived from static images for emulating more real and complicated stimuli. Despite these simple scenarios effectively help to separate key factors influencing visual coding, complex temporal relationships receive no consideration. To decompose the temporal features of visual coding in natural scenes, here we propose Vi-ST, a spatiotemporal convolutional neural network fed with a self-supervised Vision Transformer (ViT) prior, aimed at unraveling the temporal-based encoding patterns of retinal neuronal populations. The model demonstrates robust predictive performance in generalization tests. Furthermore, through detailed ablation experiments, we demonstrate the significance of each temporal module. Furthermore, we introduce a visual coding evaluation metric designed to integrate temporal considerations and compare the impact of different numbers of neuronal populations on complementary coding. In conclusion, our proposed Vi-ST demonstrates a novel modeling framework for neuronal coding of dynamic visual scenes in the brain, effectively aligning our brain representation of video with neuronal activity. The code is available at <a class="link-external link-https" href="https://github.com/wurining/Vi-ST" rel="external noopener nofollow">this https URL</a>.

A Temporal Encoding Method Based on Expansion Representation

Sparse Temporal Encoding of Visual Features for Robust Object Recognition by Spiking Neurons

A Brain-Inspired Spiking Neural Network Model with Temporal Encoding and Learning

Pattern Recognition Computation in a Spiking Neural Network with Temporal Encoding and Learning

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Rapid Feedforward Computation by Temporal Encoding and Learning with Spiking Neurons

A Spiking Neural Network System for Robust Sequence Recognition

Spiking-timing based pattern recognition with real-world visual stimuli

Robust Transcoding Sensory Information with Neural Spikes

A Spike-Timing Based Integrated Model for Pattern Recognition

Temporal Neural Encoding Methods for Spiking Neural Networks

An Event-Driven Computational System With Spiking Neurons For Object Recognition

Learning Real-World Stimuli By Single-Spike Coding And Tempotron Rule

Temporal Spiking Generative Adversarial Networks for Heading Direction Decoding

When Spiking neural networks meet temporal attention image decoding and adaptive spiking neuron

A Neurally Inspired Pattern Recognition Approach with Latency-Phase Encoding and Precise-Spike-driven Rule in Spiking Neural Network.

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

VTSNN: a virtual temporal spiking neural network

Aligning Neuronal Coding of Dynamic Visual Scenes with Foundation Vision Models

Neural Population Coding for Effective Temporal Classification