Abstract:Recent advances in spiking neural networks (SNNs) have a predominant focus on network architectures, while relatively little attention has been paid to the underlying neuron model. The point neuron models, a cornerstone of deep SNNs, pose a bottleneck on the network-level expressivity since they depict somatic dynamics only. In contrast, the multi-compartment models in neuroscience offer remarkable expressivity by introducing dendritic morphology and dynamics, but remain underexplored in deep learning due to their unaffordable computational cost and inflexibility. To combine the advantages of both sides for a flexible, efficient yet more powerful model, we propose the dendritic spiking neuron (DendSN) incorporating multiple dendritic branches with nonlinear dynamics. Compared to the point spiking neurons, DendSN exhibits significantly higher expressivity. DendSN's flexibility enables its seamless integration into diverse deep SNN architectures. To accelerate dendritic SNNs (DendSNNs), we parallelize dendritic state updates across time steps, and develop Triton kernels for GPU-level acceleration. As a result, we can construct large-scale DendSNNs with depth comparable to their point SNN counterparts. Next, we comprehensively evaluate DendSNNs' performance on various demanding tasks. By modulating dendritic branch strengths using a context signal, catastrophic forgetting of DendSNNs is substantially mitigated. Moreover, DendSNNs demonstrate enhanced robustness against noise and adversarial attacks compared to point SNNs, and excel in few-shot learning settings. Our work firstly demonstrates the possibility of training bio-plausible dendritic SNNs with depths and scales comparable to traditional point SNNs, and reveals superior expressivity and robustness of reduced dendritic neuron models in deep learning, thereby offering a fresh perspective on advancing neural network design.

Parameter efficient dendritic-tree neurons outperform perceptrons

Dendrites endow artificial neural networks with accurate, robust and parameter-efficient learning

Can Single Neurons Solve MNIST? The Computational Power of Biological Dendritic Trees

Computational and Storage Efficient Quadratic Neurons for Deep Neural Networks

Power-efficient neural network with artificial dendrites

Drawing Inspiration from Biological Dendrites to Empower Artificial Neural Networks

Dual-mode Dendritic Devices Enhanced Neural Network Based on Electrolyte Gated Transistors

Efficient dendritic learning as an alternative to synaptic plasticity hypothesis

Flexible and Scalable Deep Dendritic Spiking Neural Networks with Multiple Nonlinear Branching

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Improving energy efficiency and classification accuracy of neuromorphic chips by learning binary synaptic crossbars

Impact of dendritic non-linearities on the computational capabilities of neurons

Discovering Long-Term Effects on Parameter Efficient Fine-tuning

Leveraging dendritic properties to advance machine learning and neuro-inspired computing

Mitigating Communication Costs in Neural Networks: The Role of Dendritic Nonlinearity

Parameter Prediction for Unseen Deep Architectures

A multi-in and multi-out dendritic neuron model and its optimization

The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks

Non-linear Neurons with Human-like Apical Dendrite Activations

Beyond spiking networks: The computational advantages of dendritic amplification and input segregation