Abstract:Objective. Neural decoding is an important tool in neural engineering and neural data analysis. Of various machine learning algorithms adopted for neural decoding, the recently introduced deep learning is promising to excel. Therefore, we sought to apply deep learning to decode movement trajectories from the activity of motor cortical neurons. Approach. In this paper, we assessed the performance of deep learning methods in three different decoding schemes, concurrent, time-delay, and spatiotemporal. In the concurrent decoding scheme where the input to the network is the neural activity coincidental to the movement, deep learning networks including artificial neural network (ANN) and long-short term memory (LSTM) were applied to decode movement and compared with traditional machine learning algorithms. Both ANN and LSTM were further evaluated in the time-delay decoding scheme in which temporal delays are allowed between neural signals and movements. Lastly, in the spatiotemporal decoding scheme, we trained convolutional neural network (CNN) to extract movement information from images representing the spatial arrangement of neurons, their activity, and connectomes (i.e. the relative strengths of connectivity between neurons) and combined CNN and ANN to develop a hybrid spatiotemporal network. To reveal the input features of the CNN in the hybrid network that deep learning discovered for movement decoding, we performed a sensitivity analysis and identified specific regions in the spatial domain. Main results. Deep learning networks (ANN and LSTM) outperformed traditional machine learning algorithms in the concurrent decoding scheme. The results of ANN and LSTM in the time-delay decoding scheme showed that including neural data from time points preceding movement enabled decoders to perform more robustly when the temporal relationship between the neural activity and movement dynamically changes over time. In the spatiotemporal decoding scheme, the hybrid spatiotemporal network containing the concurrent ANN decoder outperformed single-network concurrent decoders. Significance. Taken together, our study demonstrates that deep learning could become a robust and effective method for the neural decoding of behavior.

Deep learning based decoding of single local field potential events

Deep Learning for Micro-Electrocorticographic (µecog) Data

Neural Decoding Using Local Field Potential Based on Partial Least Squares Regression.

Dimensionality Reduction of Local Field Potential Features with Convolution Neural Network in Neural Decoding: A Pilot Study

A Hybrid Autoencoder Framework of Dimensionality Reduction for Brain-Computer Interface Decoding

Deep learning approaches for neural decoding across architectures and recording modalities

Long-term Decoding Stability of Local Field Potentials from Silicon Arrays in Primate Motor Cortex During a 2D Center out Task

Deep learning for neural decoding in motor cortex

Decoding Forelimb Muscle Activity with Local Field Potentials from Mouse Motor Cortex

The comparison of long-term decoding performance between low and high frequency local field potentials in macaque monkeys

Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI

Deep learning with convolutional neural networks for decoding and visualization of EEG pathology

Deep Learning Decoding of Mental State in Non-invasive Brain Computer Interface

Deep learning with convolutional neural networks for EEG decoding and visualization

Decoding Working Memory Representation Based on Single-Trial EEG by a Deep Learning Framework

Decoding Brain States Based on Microcircuits

Inferring single-trial neural population dynamics using sequential auto-encoders

Group-level Brain Decoding with Deep Learning

End-to-end multimodal deep learning for real-time decoding of months-long neural activity from the same cells

Unsupervised decoding of long-term, naturalistic human neural recordings with automated video and audio annotations

Interpretable Deep Neural Networks for Single-Trial EEG Classification