Theories and Methods for Large-Scale Brain-Inspired Neural Networks
Zhengyu Ma,Yonghong Tian
DOI: https://doi.org/10.1360/tb-2023-0775
2023-01-01
Chinese Science Bulletin
Abstract:The brain's structure and functions have consistently inspired the development of intelligent technologies throughout human history.Brain-inspired spiking neural networks(SNNs)emulate neural dynamics by incorporating event-based computations to capture temporal information,thereby enabling brain-like and energy-efficient computations.Currently,SNNs have been successfully applied in various fields,such as image recognition and object detection,and have the potential to revolutionize the field of artificial intelligence.Similar to the brain,artificial models show emergent properties,such as high performance and intelligence,with sufficiently large size.However,the sizes of SNNs have been limited,which hinders potential performance improvements and thus their real-world application.Therefore,methods and theories for constructing large-scale SNNs must be developed to achieve their highest potential.New techniques for building large-scale SNNs have recently attracted extensive research interest.In this review,notable advances and developments with large-scale SNNs,including spiking neuron,algorithm and model structure designs,software frameworks and neuromorphic computing chips,and neuromorphic applications are summarized.Various strategies including parallel computing,surrogate gradients,and transformer-based SNNs are discussed in detail.Specifically,various types of spiking neurons have been developed,with parametric neurons improving the representation ability of SNNs and parallel spiking neurons accelerating the training process.The direct training method is notable for enabling the training of large-scale SNNs,while brain-inspired methods such as plasticity,attention,and pruning algorithms improve the robustness and energy efficiency of SNNs.The ResNet structure and its variants have led to the development of large-scale SNNs,and the emergence of transformer-based SNNs has significantly improved SNN performance on computer vision tasks.Programming frameworks have evolved from solely supporting the inference of large-scale networks to facilitating training,including automatic differentiation,parallel computation acceleration,and implementation on neuromorphic sensors and chips.Large SNNs are particularly well-suited for processing events/spikes collected by neuromorphic sensors,especially in image reconstruction and object detection,but their development is still in the early stages.Furthermore,existing challenges and deficiencies are discussed,providing guidance for the future development of large-scale SNN models.In summary,brain-inspired designs of models and neuromorphic chips are still limited to simplified simulations,lacking sufficient biological plausibility.The temporal dynamic representation capabilities of SNNs have not yet been fully demonstrated.The computations of large-scale SNNs must be accelerated while reducing their power consumption.Moreover,tasks such as natural language processing,reasoning,and decision-making and computer vision tasks must both be investigated.The problem of efficiently measuring and representing asynchronous spatiotemporal events/spike data remains unresolved,limiting the full potential of existing neuromorphic vision systems.In conclusion,large-scale brain-inspired neural networks can still be substantially improved.Efforts are needed in various research directions,including algorithms,computing frameworks,and architectures,to explore and optimize brain-like models and hardware in a collaborative manner.