Abstract:Individual neurons in the brain have complex intrinsic dynamics that are highly diverse. We hypothesize that the complex dynamics produced by networks of complex and heterogeneous neurons may contribute to the brain's ability to process and respond to temporally complex data. To study the role of complex and heterogeneous neuronal dynamics in network computation, we develop a rate-based neuronal model, the generalized-leaky-integrate-and-fire-rate (GLIFR) model, which is a rate equivalent of the generalized-leaky-integrate-and-fire model. The GLIFR model has multiple dynamical mechanisms, which add to the complexity of its activity while maintaining differentiability. We focus on the role of after-spike currents, currents induced or modulated by neuronal spikes, in producing rich temporal dynamics. We use machine learning techniques to learn both synaptic weights and parameters underlying intrinsic dynamics to solve temporal tasks. The GLIFR model allows the use of standard gradient descent techniques rather than surrogate gradient descent, which has been used in spiking neural networks. After establishing the ability to optimize parameters using gradient descent in single neurons, we ask how networks of GLIFR neurons learn and perform on temporally challenging tasks, such as sequential MNIST. We find that these networks learn diverse parameters, which gives rise to diversity in neuronal dynamics, as demonstrated by clustering of neuronal parameters. GLIFR networks have mixed performance when compared to vanilla recurrent neural networks, with higher performance in pixel-by-pixel MNIST but lower in line-by-line MNIST. However, they appear to be more robust to random silencing. We find that the ability to learn heterogeneity and the presence of after-spike currents contribute to these gains in performance. Our work demonstrates both the computational robustness of neuronal complexity and diversity in networks and a feasible method of training such models using exact gradients.

Neuronal diversity can improve machine learning for physics and beyond

Neural diversity is key to collective artificial learning

Dive into the Power of Neuronal Heterogeneity

Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networks

Enhancing learning in spiking neural networks through neuronal heterogeneity and neuromodulatory signaling

On Diversity in Discriminative Neural Networks

Heterogeneity in Neuronal Dynamics Is Learned by Gradient Descent for Temporal Processing Tasks

Growing Artificial Neural Networks for Control: the Role of Neuronal Diversity

Robust Computation with Intrinsic Heterogeneity

Understanding Neuronal Diversity: Role of Input Dynamics and Selectivity

Neural Heterogeneity Enhances Reliable Neural information Processing: Local Sensitivity and Globally Input-slaved Transient Dynamics

Diversity-induced trivialization and resilience of neural dynamics

Kernel Heterogeneity Improves Sparseness of Natural Images Representations

Amplified Signal Response by Neuronal Diversity on Complex Networks

Neuronal firing rate diversity lowers the dimension of population covariability

Biologically Inspired Heterogeneous Learning for Accurate, Efficient and Low-Latency Neural Network

Enhancement of Neuronal Coherence by Diversity in Coupled Rulkov-Map Models

Diversity in Machine Learning

Heterogeneous Neuronal and Synaptic Dynamics for Spike-Efficient Unsupervised Learning: Theory and Design Principles

Effect Exploration of the Spiking Neuron Property and Heterogeneity on Brain-Inspired Context-Dependent Learning

Effect of Synaptic Heterogeneity on Neuronal Coordination