William F. Podlaski,Christian K. Machens
Abstract:Associative memory architectures such as the Hopfield network have long been important conceptual and theoretical models for neuroscience and artificial intelligence. However, translating these abstract models into spiking neural networks has been surprisingly difficult. Indeed, much previous work has been restricted to storing a small number of primarily non-overlapping memories in large networks, thereby limiting their scalability. Here, we revisit the associative memory problem in light of recent advances in understanding spike-based computation. Using a recently-established geometric framework, we show that the spiking activity for a large class of all-inhibitory networks is situated on a low-dimensional, convex, and piecewise-linear manifold, with dynamics that move along the manifold. We then map the associative memory problem onto these dynamics, and demonstrate how the vertices of a hypercubic manifold can be used to store stable, overlapping activity patterns with a direct correspondence to the original Hopfield model. We propose several learning rules, and demonstrate a linear scaling of the storage capacity with the number of neurons, as well as robust pattern completion abilities. Overall, this work serves as a case study to demonstrate the effectiveness of using a geometrical perspective to design dynamics on neural manifolds, with implications for neuroscience and machine learning.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to store a large number of overlapping associative memories in low - rank spiking neural networks. Specifically, traditionally, the main challenge in implementing associative memory models (such as Hopfield networks) in spiking neural networks is that only a small number of mostly non - overlapping memory patterns can be stored in large networks, which limits the scalability and practical application capabilities of these models. By introducing a geometric framework, this paper re - examines the associative memory problem and proposes a new method to design and control network dynamics, enabling the storage of stable, overlapping activity patterns on low - dimensional convex polyhedral manifolds, thereby increasing the storage capacity and pattern completion ability.
### Main contributions of the paper:
1. **Introduction of the geometric framework**: The authors use a recently established geometric framework to show that the spiking activities of most inhibitory networks are located on a low - dimensional, convex, piecewise - linear manifold, and their dynamics move along the manifold.
2. **Utilization of the vertices of the hypercube**: By mapping the associative memory problem onto these dynamics, the authors prove that the vertices of the hypercube manifold can be used to store stable, overlapping activity patterns that directly correspond to the original Hopfield model.
3. **Design of learning rules**: Several learning rules are proposed, including the Hebbian rule, the pseudo - inverse rule, and the optimization rule, demonstrating the effectiveness of these rules in terms of storage capacity and pattern completion.
4. **Linearly scalable storage capacity**: Experimental results show that the storage capacity has a linear relationship with the number of neurons, while having strong pattern completion ability.
### Formula explanations:
- **Voltage dynamics**:
\[
\dot{V}(t)=-V(t)+W s(t)+c(t)
\]
where \(V(t)\) is the membrane potential, \(W\) is the recurrent connection weight, \(s(t)\) is the synaptic input or spike, and \(c(t)\) is the external input.
- **Filtered voltage**:
\[
V(t)=W r(t)+I_{\text{ext}}(t)
\]
where \(r(t)\) is the filtered spike, and \(I_{\text{ext}}(t)\) is the filtered external input.
- **Rank - constrained decomposition**:
\[
W = ED
\]
where \(E\in\mathbb{R}^{N\times K}\) and \(D\in\mathbb{R}^{K\times N}\) are the encoding matrix and the decoding matrix, respectively.
- **Latent variable**:
\[
y(t)=D r(t)
\]
- **Vertex fixed - point stability**:
\[
D\eta_\mu=\xi_\mu,\quad\forall\mu
\]
where \(\eta_\mu\) is the activity pattern in the neural space, and \(\xi_\mu\) is the pattern in the latent space.
- **Optimization learning rule**:
\[
D_{\text{opt}}=\arg\min_D\|D\|_2^2\quad\text{subject to}\quad D\eta = \xi
\]
### Conclusion:
By introducing the geometric framework, this paper successfully realizes the storage of a large number of overlapping associative memories in low - rank spiking neural networks, solving the problems of limited storage capacity and pattern overlap in traditional methods. This work not only provides a new perspective for neuroscience and machine learning but also shows how to use geometric methods to design and control the dynamics of neural networks.