Abstract:Objective. Neural decoding is an important tool in neural engineering and neural data analysis. Of various machine learning algorithms adopted for neural decoding, the recently introduced deep learning is promising to excel. Therefore, we sought to apply deep learning to decode movement trajectories from the activity of motor cortical neurons. Approach. In this paper, we assessed the performance of deep learning methods in three different decoding schemes, concurrent, time-delay, and spatiotemporal. In the concurrent decoding scheme where the input to the network is the neural activity coincidental to the movement, deep learning networks including artificial neural network (ANN) and long-short term memory (LSTM) were applied to decode movement and compared with traditional machine learning algorithms. Both ANN and LSTM were further evaluated in the time-delay decoding scheme in which temporal delays are allowed between neural signals and movements. Lastly, in the spatiotemporal decoding scheme, we trained convolutional neural network (CNN) to extract movement information from images representing the spatial arrangement of neurons, their activity, and connectomes (i.e. the relative strengths of connectivity between neurons) and combined CNN and ANN to develop a hybrid spatiotemporal network. To reveal the input features of the CNN in the hybrid network that deep learning discovered for movement decoding, we performed a sensitivity analysis and identified specific regions in the spatial domain. Main results. Deep learning networks (ANN and LSTM) outperformed traditional machine learning algorithms in the concurrent decoding scheme. The results of ANN and LSTM in the time-delay decoding scheme showed that including neural data from time points preceding movement enabled decoders to perform more robustly when the temporal relationship between the neural activity and movement dynamically changes over time. In the spatiotemporal decoding scheme, the hybrid spatiotemporal network containing the concurrent ANN decoder outperformed single-network concurrent decoders. Significance. Taken together, our study demonstrates that deep learning could become a robust and effective method for the neural decoding of behavior.

Neural Decoding on Imbalanced Calcium Imaging Data with a Network of Support Vector Machines

Real-Time Calcium Imaging Based Neural Decoding with a Support Vector Machine.

Robust neural decoding by kernel regression with Siamese representation learning

Online Decoding System with Calcium Image from Mice Primary Motor Cortex.

Classification of A Pressing Movement by Background Removed Single- photon Calcium Image Sequences

Continuous Prediction of Mice Lever-Pressing Kinematic Parameters by Background Removed Single-photon Calcium Images

Neural Decoding Based on Probabilistic Neural Network

Real-Time Neuron Detection and Neural Signal Extraction Platform for Miniature Calcium Imaging.

Dimensionality Reduction of Local Field Potential Features with Convolution Neural Network in Neural Decoding: A Pilot Study

Neural Decoding Using a Parallel Sequential Monte Carlo Method on Point Processes with Ensemble Effect.

Somatic Calcium Signals from Layer II/III Motor Cortex for Continuous Neural Decoding

Deep learning for neural decoding in motor cortex

An end-to-end recurrent compressed sensing method to denoise, detect and demix calcium imaging data

Jointly Fusing Multi-Scale Spatial-Logical Brain Networks: A Neural Decoding Method

FPGA-Based In-Vivo Calcium Image Decoding for Closed-Loop Feedback Applications

Penalized matrix decomposition for denoising, compression, and improved demixing of functional imaging data

Decoding multi-limb movements from two-photon calcium imaging of neuronal activity using deep learning

Fast amortized inference of neural activity from calcium imaging data with variational autoencoders

Nonsmooth Formulation of the Support Vector Machine for a Neural Decoding Problem

Reinforcing Neuron Extraction and Spike Inference in Calcium Imaging Using Deep Self-Supervised Denoising

ViF-SD2E: A Robust Weakly-Supervised Method for Neural Decoding