Abstract:Neural encoding, a crucial aspect to understand the human brain information processing system, aims to establish a quantitative relationship between the stimuli and the evoked brain activities. In the field of visual neuroscience, with the ability to explain how neurons in the primary visual cortex work, population receptive field (pRF) models have enjoyed high popularity and made reliable progress in recent years. However, existing models rely on either the inflexible prior assumptions about pRF or the clumsy parameter estimation methods, severely limiting the expressiveness and interpretability. In this article, we propose a novel neural encoding framework by learning "what" and "where" with deep neural networks. It involves two separate aspects: 1) the spatial characteristic ("where") and 2) feature selection ("what") of neuron populations in the visual cortex. Specifically, our approach first encodes visual stimuli into hierarchically intermediate features through a pretrained deep neural network (DNN), then converts DNN features into refined features with the channel attention and spatial receptive field (RF) to learn "where", and finally regresses refined features simultaneously onto voxel activities to learn "what". The sparsity regularization and smoothness regularization are adopted in our modeling approach so that the crucial RF can be estimated automatically without prior assumptions about shapes. Furthermore, an attempt is made to extend the voxel-wise modeling approach to multi-voxel joint encoding models, and we show that it is conducive to rescuing voxels with poor signal-to-noise characteristics. Extensive empirical results demonstrate that the method developed herein provides an effective strategy to establish neural encodin- for the human visual cortex, with the weaker prior constraints but the higher encoding performance.

Neural encoding with unsupervised spiking convolutional neural network

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Robust Transcoding Sensory Information with Neural Spikes

A Brain-Inspired Spiking Neural Network Model with Temporal Encoding and Learning

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Motorsrnn: A Spiking Recurrent Neural Network Inspired by Brain Topology for the Effective and Efficient Decoding of Cortical Spike Trains

Rapid Feedforward Computation by Temporal Encoding and Learning with Spiking Neurons

A Visual Encoding Model Based on Deep Neural Networks and Transfer Learning for Brain Activity Measured by Functional Magnetic Resonance Imaging

Pattern Recognition Computation in a Spiking Neural Network with Temporal Encoding and Learning

Emergent Bio-Functional Similarities in a Cortical-Spike-Train-Decoding Spiking Neural Network Facilitate Predictions of Neural Computation

Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning “What” and “Where”

A visual encoding model based on deep neural networks and transfer learning

VTSNN: a virtual temporal spiking neural network

A Spatial–Channel–Temporal-Fused Attention for Spiking Neural Networks

STSC-SNN: Spatio-Temporal Synaptic Connection with temporal convolution and attention for spiking neural networks

A Novel Spiking Neural Network of Receptive Field Encoding with Groups of Neurons Decision

Unsupervised Spiking Neural Network with Dynamic Learning of Inhibitory Neurons

Unsupervised Multi-layer Spiking Convolutional Neural Network Using Layer-Wise Sparse Coding.

A Spiking Neural Network With Adaptive Graph Convolution and LSTM for EEG-Based Brain-Computer Interfaces

High-parallelism Inception-like Spiking Neural Networks for Unsupervised Feature Learning