Abstract:<p>Neural coding is one of the central questions in systems neuroscience for understanding how the brain processes stimulus from the environment, moreover, it is also a cornerstone for designing algorithms of brain-machine interface, where decoding incoming stimulus is highly demanded for better performance of physical devices. Traditionally researchers have focused on functional magnetic resonance imaging (fMRI) data as the neural signals of interest for decoding visual scenes. However, our visual perception operates in a fast time scale of millisecond in terms of an event termed neural spike. There are few studies of decoding by using spikes. Here we fulfill this aim by developing a novel decoding framework based on deep neural networks, named spike-image decoder (SID), for reconstructing natural visual scenes, including static images and dynamic videos, from experimentally recorded spikes of a population of retinal ganglion cells. The SID is an end-to-end decoder with one end as neural spikes and the other end as images, which can be trained directly such that visual scenes are reconstructed from spikes in a highly accurate fashion. Our SID also outperforms on the reconstruction of visual stimulus compared to existing fMRI decoding models. In addition, with the aid of a spike encoder, we show that SID can be generalized to arbitrary visual scenes by using the image datasets of MNIST, CIFAR10, and CIFAR100. Furthermore, with a pre-trained SID, one can decode any dynamic videos to achieve real-time encoding and decoding of visual scenes by spikes. Altogether, our results shed new light on neuromorphic computing for artificial visual systems, such as event-based visual cameras and visual neuroprostheses.</p>

RNCIR: Retinal Neuron Coding-Based Image Recognition

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

A Retina-LGN-V1 Structure-like Spiking Neuron Network for Image Feature Extraction

Application Of Precise-Spike-Driven Rule In Spiking Neural Networks For Optical Character Recognition

Sparse Temporal Encoding of Visual Features for Robust Object Recognition by Spiking Neurons

A Temporal Encoding Method Based on Expansion Representation

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Robust Transcoding Sensory Information with Neural Spikes

A Brain-Inspired Spiking Neural Network Model with Temporal Encoding and Learning

A Neurally Inspired Pattern Recognition Approach with Latency-Phase Encoding and Precise-Spike-driven Rule in Spiking Neural Network.

Motorsrnn: A Spiking Recurrent Neural Network Inspired by Brain Topology for the Effective and Efficient Decoding of Cortical Spike Trains

An Event-Driven Computational System With Spiking Neurons For Object Recognition

Pattern Recognition of Spiking Neural Networks Based on Visual Mechanism and Supervised Synaptic Learning

Pattern Recognition Computation in a Spiking Neural Network with Temporal Encoding and Learning

NatCSNN: A Convolutional Spiking Neural Network for recognition of objects extracted from natural images

An Image Recognizing Method Based on Precise Moment of Spikes

BioLCNet: Reward-modulated Locally Connected Spiking Neural Networks

Direct training high-performance spiking neural networks for object recognition and detection

Neural encoding with unsupervised spiking convolutional neural network

Reconstruction of natural visual scenes from neural spikes with deep neural networks

EEG-Based Visual Classification with Recurrent Neural Network