Abstract:Individual neurons in the brain have complex intrinsic dynamics that are highly diverse. We hypothesize that the complex dynamics produced by networks of complex and heterogeneous neurons may contribute to the brain's ability to process and respond to temporally complex data. To study the role of complex and heterogeneous neuronal dynamics in network computation, we develop a rate-based neuronal model, the generalized-leaky-integrate-and-fire-rate (GLIFR) model, which is a rate equivalent of the generalized-leaky-integrate-and-fire model. The GLIFR model has multiple dynamical mechanisms, which add to the complexity of its activity while maintaining differentiability. We focus on the role of after-spike currents, currents induced or modulated by neuronal spikes, in producing rich temporal dynamics. We use machine learning techniques to learn both synaptic weights and parameters underlying intrinsic dynamics to solve temporal tasks. The GLIFR model allows the use of standard gradient descent techniques rather than surrogate gradient descent, which has been used in spiking neural networks. After establishing the ability to optimize parameters using gradient descent in single neurons, we ask how networks of GLIFR neurons learn and perform on temporally challenging tasks, such as sequential MNIST. We find that these networks learn diverse parameters, which gives rise to diversity in neuronal dynamics, as demonstrated by clustering of neuronal parameters. GLIFR networks have mixed performance when compared to vanilla recurrent neural networks, with higher performance in pixel-by-pixel MNIST but lower in line-by-line MNIST. However, they appear to be more robust to random silencing. We find that the ability to learn heterogeneity and the presence of after-spike currents contribute to these gains in performance. Our work demonstrates both the computational robustness of neuronal complexity and diversity in networks and a feasible method of training such models using exact gradients.

Inhomogeneous metallic phase in a disordered Mott insulator in two dimensions.

Context Gating in Spiking Neural Networks: Achieving Lifelong Learning through Integration of Local and Global Plasticity

Learning cortical hierarchies with temporal Hebbian updates

Synergistic pathways of modulation enable robust task packing within neural dynamics

Flexible task abstractions emerge in linear networks with fast and bounded units

Unleashing the Potential of Spiking Neural Networks for Sequential Modeling with Contextual Embedding.

Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization

NMDA-driven dendritic modulation enables multitask representation learning in hierarchical sensory processing pathways

Parallel Learning by Multitasking Neural Networks

Neuron-centric Hebbian Learning

Towards continual task learning in artificial neural networks: current approaches and insights from neuroscience

Learning to Modulate Random Weights: Neuromodulation-inspired Neural Networks For Efficient Continual Learning

Heterogeneity in Neuronal Dynamics Is Learned by Gradient Descent for Temporal Processing Tasks

Do simple sequence repeats in replication, repair and recombination genes of mycoplasmas provide genetic variability?

Efficient Weight-Space Laplace-Gaussian Filtering and Smoothing for Sequential Deep Learning

Beyond spiking networks: The computational advantages of dendritic amplification and input segregation

Disentangling and Mitigating the Impact of Task Similarity for Continual Learning

Bio-inspired, task-free continual learning through activity regularization

One Step Back, Two Steps Forward: Interference and Learning in Recurrent Neural Networks

Keep Moving: identifying task-relevant subspaces to maximise plasticity for newly learned tasks

Flexible gating between subspaces in a neural network model of internally guided task switching