Abstract:<p>Neural coding is one of the central questions in systems neuroscience for understanding how the brain processes stimulus from the environment, moreover, it is also a cornerstone for designing algorithms of brain-machine interface, where decoding incoming stimulus is highly demanded for better performance of physical devices. Traditionally researchers have focused on functional magnetic resonance imaging (fMRI) data as the neural signals of interest for decoding visual scenes. However, our visual perception operates in a fast time scale of millisecond in terms of an event termed neural spike. There are few studies of decoding by using spikes. Here we fulfill this aim by developing a novel decoding framework based on deep neural networks, named spike-image decoder (SID), for reconstructing natural visual scenes, including static images and dynamic videos, from experimentally recorded spikes of a population of retinal ganglion cells. The SID is an end-to-end decoder with one end as neural spikes and the other end as images, which can be trained directly such that visual scenes are reconstructed from spikes in a highly accurate fashion. Our SID also outperforms on the reconstruction of visual stimulus compared to existing fMRI decoding models. In addition, with the aid of a spike encoder, we show that SID can be generalized to arbitrary visual scenes by using the image datasets of MNIST, CIFAR10, and CIFAR100. Furthermore, with a pre-trained SID, one can decode any dynamic videos to achieve real-time encoding and decoding of visual scenes by spikes. Altogether, our results shed new light on neuromorphic computing for artificial visual systems, such as event-based visual cameras and visual neuroprostheses.</p>

Retina-Like Visual Image Reconstruction Via Spiking Neural Model

Ultra-high Temporal Resolution Visual Reconstruction from a Fovea-like Spike Camera via Spiking Neuron Model

High-speed Image Reconstruction through Short-term Plasticity for Spiking Cameras

Reconstructing Clear Image for High-Speed Motion Scene With a Retina-Inspired Spike Camera

NeuSpike-Net: High Speed Video Reconstruction via Bio-inspired Neuromorphic Cameras

Capture the Moment: High-Speed Imaging With Spiking Cameras Through Short-Term Plasticity

Spk2ImgNet: Learning to Reconstruct Dynamic Scene from Continuous Spike Stream

Spike Camera Image Reconstruction Using Deep Spiking Neural Networks

High-Speed Motion Scene Reconstruction for Spike Camera via Motion Aligned Filtering

High-speed and High-quality Vision Reconstruction of Spike Camera with Spike Stability Theorem

A Retina-inspired Sampling Method for Visual Texture Reconstruction

Review of Visual Reconstruction Methods of Retina-Like Vision Sensors

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

A Multi-Scale Threshold Spiking Mechanism Based on Spike Camera

Boosting Spike Camera Image Reconstruction from a Perspective of Dealing with Spike Fluctuations

A Robust Visual Sampling Model Inspired by Receptive Field

Spike-NeRF: Neural Radiance Field Based On Spike Camera

Reconstruction of natural visual scenes from neural spikes with deep neural networks

Spike Signal Reconstruction Based on Inter-Spike Similarity

Self-Supervised Mutual Learning for Dynamic Scene Reconstruction of Spiking Camera.

Recurrent Spike-based Image Restoration under General Illumination