Abstract:We propose a two-stage memory retrieval dynamics for modern Hopfield models, termed $\mathtt{U\text{-}Hop}$, with enhanced memory capacity. Our key contribution is a learnable feature map $\Phi$ which transforms the Hopfield energy function into kernel space. This transformation ensures convergence between the local minima of energy and the fixed points of retrieval dynamics within the kernel space. Consequently, the kernel norm induced by $\Phi$ serves as a novel similarity measure. It utilizes the stored memory patterns as learning data to enhance memory capacity across all modern Hopfield models. Specifically, we accomplish this by constructing a separation loss $\mathcal{L}_\Phi$ that separates the local minima of kernelized energy by separating stored memory patterns in kernel space. Methodologically, $\mathtt{U\text{-}Hop}$ memory retrieval process consists of: (Stage I) minimizing separation loss for a more uniform memory (local minimum) distribution, followed by (Stage II) standard Hopfield energy minimization for memory retrieval. This results in a significant reduction of possible metastable states in the Hopfield energy function, thus enhancing memory capacity by preventing memory confusion. Empirically, with real-world datasets, we demonstrate that $\mathtt{U\text{-}Hop}$ outperforms all existing modern Hopfield models and state-of-the-art similarity measures, achieving substantial improvements in both associative memory retrieval and deep learning tasks. Code is available at <a class="link-external link-https" href="https://github.com/MAGICS-LAB/UHop" rel="external noopener nofollow">this https URL</a> ; future updates are on <a class="link-https" data-arxiv-id="2404.03827" href="https://arxiv.org/abs/2404.03827">arXiv:2404.03827</a>

On the Critical Capacity of the Hopfield Model

Comment on ``Capacity of the Hopfield model''

On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysis

Analysis of the Hopfield Model with Discrete Coupling

The capacity of the Hopfield associative memory

Statistical Mechanics of a Hopfield Neural-Network Model in a Transverse Field

Dynamic Capacity Estimation in Hopfield Networks

Statistical Mechanics of a Multiconnected Hopfield Neural-Network Model in a Transverse Field

Storage capacity of the Hopfield neural network

Effect of spatial correlations on Hopfield Neural Network and Dense Associative Memories

High storage capacity in the Hopfield model with auto-interactions - stability analysis

Quantum Hopfield model with a random transverse field and a random neuronal threshold

Tricritical Points and Reentry in the Quantum Hopfield Neural-Network Model

Storage Capacity of the Hopfield Network Associative Memory

Exact Replica Symmetric solution for transverse field Hopfield model under finite Trotter size

A new mechanical approach to handle generalized Hopfield neural networks

Reconstructing the Hopfield network as an inverse Ising problem

Bounding the Graph Capacity with Quantum Mechanics and Finite Automata

The thermodynamic limit in mean field neural networks

Out-of-equilibrium dynamics of the Hopfield model in its spin-glass phase

Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models