Abstract:Individual neurons in the brain have complex intrinsic dynamics that are highly diverse. We hypothesize that the complex dynamics produced by networks of complex and heterogeneous neurons may contribute to the brain's ability to process and respond to temporally complex data. To study the role of complex and heterogeneous neuronal dynamics in network computation, we develop a rate-based neuronal model, the generalized-leaky-integrate-and-fire-rate (GLIFR) model, which is a rate equivalent of the generalized-leaky-integrate-and-fire model. The GLIFR model has multiple dynamical mechanisms, which add to the complexity of its activity while maintaining differentiability. We focus on the role of after-spike currents, currents induced or modulated by neuronal spikes, in producing rich temporal dynamics. We use machine learning techniques to learn both synaptic weights and parameters underlying intrinsic dynamics to solve temporal tasks. The GLIFR model allows the use of standard gradient descent techniques rather than surrogate gradient descent, which has been used in spiking neural networks. After establishing the ability to optimize parameters using gradient descent in single neurons, we ask how networks of GLIFR neurons learn and perform on temporally challenging tasks, such as sequential MNIST. We find that these networks learn diverse parameters, which gives rise to diversity in neuronal dynamics, as demonstrated by clustering of neuronal parameters. GLIFR networks have mixed performance when compared to vanilla recurrent neural networks, with higher performance in pixel-by-pixel MNIST but lower in line-by-line MNIST. However, they appear to be more robust to random silencing. We find that the ability to learn heterogeneity and the presence of after-spike currents contribute to these gains in performance. Our work demonstrates both the computational robustness of neuronal complexity and diversity in networks and a feasible method of training such models using exact gradients.

Robust Computation with Intrinsic Heterogeneity

Firing Rate Dynamics in Recurrent Spiking Neural Networks with Intrinsic and Network Heterogeneity

Dive into the Power of Neuronal Heterogeneity

Neural Heterogeneity Enhances Reliable Neural information Processing: Local Sensitivity and Globally Input-slaved Transient Dynamics

Heterogeneity in Neuronal Dynamics Is Learned by Gradient Descent for Temporal Processing Tasks

Effect of Synaptic Heterogeneity on Neuronal Coordination

Heterogeneous and higher-order cortical connectivity undergirds efficient, robust and reliable neural codes

Neuronal diversity can improve machine learning for physics and beyond

Stable recurrent dynamics in heterogeneous neuromorphic computing systems using excitatory and inhibitory plasticity

Kernel Heterogeneity Improves Sparseness of Natural Images Representations

Biologically Inspired Heterogeneous Learning for Accurate, Efficient and Low-Latency Neural Network

Variable synaptic strengths controls the firing rate distribution in feedforward neural networks

Exploiting Heterogeneity in Timescales for Sparse Recurrent Spiking Neural Networks for Energy-Efficient Edge Computing

Diversity-induced trivialization and resilience of neural dynamics

Effect of heterogeneity for the self-organization of a neural network

Enhancing learning in spiking neural networks through neuronal heterogeneity and neuromodulatory signaling

Optimizing working memory with heterogeneity of recurrent cortical excitation

Heterogeneous encoding of temporal stimuli in the cerebellar cortex

Optimal responsiveness and collective oscillations emerging from the heterogeneity of inhibitory neurons

Effect Exploration of the Spiking Neuron Property and Heterogeneity on Brain-Inspired Context-Dependent Learning

Exploiting Inherent Error-Resiliency of Neuromorphic Computing to achieve Extreme Energy-Efficiency through Mixed-Signal Neurons