Spiking representation learning for associative memories

Naresh Ravichandran,Anders Lansner,Pawel Herman
2024-06-05
Abstract:Networks of interconnected neurons communicating through spiking signals offer the bedrock of neural computations. Our brains spiking neural networks have the computational capacity to achieve complex pattern recognition and cognitive functions effortlessly. However, solving real-world problems with artificial spiking neural networks (SNNs) has proved to be difficult for a variety of reasons. Crucially, scaling SNNs to large networks and processing large-scale real-world datasets have been challenging, especially when compared to their non-spiking deep learning counterparts. The critical operation that is needed of SNNs is the ability to learn distributed representations from data and use these representations for perceptual, cognitive and memory operations. In this work, we introduce a novel SNN that performs unsupervised representation learning and associative memory operations leveraging Hebbian synaptic and activity-dependent structural plasticity coupled with neuron-units modelled as Poisson spike generators with sparse firing (~1 Hz mean and ~100 Hz maximum firing rate). Crucially, the architecture of our model derives from the neocortical columnar organization and combines feedforward projections for learning hidden representations and recurrent projections for forming associative memories. We evaluated the model on properties relevant for attractor-based associative memories such as pattern completion, perceptual rivalry, distortion resistance, and prototype extraction.
Neural and Evolutionary Computing,Neurons and Cognition
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: How to achieve the ability to learn distributed representations from data in artificial spiking neural networks (SNNs), and use these representations for perception, cognition and memory operations, especially the associative memory function. Specifically, the paper focuses on constructing an SNN model that can handle large - scale real - world data sets, which can learn sparse and distributed internal representations in an unsupervised manner, and this ability is comparable to that of non - spiking deep - learning models. ### Background and Challenges 1. **Advantages of Spiking Neural Networks (SNNs)**: - SNNs imitate neurons in the brain communicating through spiking signals and have the ability to efficiently process real - time data streams. - Compared with traditional non - spiking deep - learning models, SNNs have significant advantages in energy efficiency. 2. **Limitations of Current SNNs**: - There is a lack of effective mechanisms to learn sparse and distributed internal representations from real - world data. - It is difficult to scale to large networks and handle large - scale real - world data sets. - Existing SNNs mainly rely on supervised learning methods and lack the ability of unsupervised learning. ### Main Contributions of the Paper 1. **Proposing a New SNN Model**: - This model is based on Hebbian synaptic plasticity and activity - dependent structural plasticity, combining feed - forward and recurrent connections. - The neuron units in the model are modeled as Poisson spike generators with a sparse firing rate (about 1 Hz on average and about 100 Hz at maximum). 2. **Model Architecture**: - Feed - forward connections are used to learn hidden representations. - Recurrent connections are used to form associative memories. - The architecture of the model draws on the neocortical columnar organization, including the competition mechanisms of the functional hypercolumn module and the minicolumn module. 3. **Experimental Verification**: - Tests were carried out on the MNIST handwritten digit data set, and the performance of the model in pattern completion, perceptual competition, anti - distortion ability and prototype extraction was evaluated. - The experimental results show that the sparse - spiking model approaches or reaches the performance of non - spiking models in representation learning and associative memory functions. ### Key Technologies and Methods 1. **Hebbian - Bayesian Learning Rule**: - Use local spatio - temporal information to update synaptic strength, making the learning mechanism local, correlative and Hebbian. - Calculate short - term and long - term statistics to capture the joint information of forward and backward spike activities. 2. **Spike Activation Function**: - The membrane potential of neurons is updated according to the total synaptic input. - Use the softmax function to calculate the firing probability of neurons, introducing a soft - winner - takes - all competition mechanism. 3. **Structural Plasticity**: - Dynamically adjust the connection structure of the network through an activity - dependent rewiring algorithm, maintaining a sparse and patchy connection matrix. ### Conclusion By introducing a new SNN model, this paper successfully solves the limitations of existing SNNs in learning sparse distributed representations and handling large - scale real - world data sets. The experimental results show that this model performs excellently in unsupervised learning and associative memory functions, providing an important reference for the development of future brain - inspired computing systems.