Abstract:We propose a two-stage memory retrieval dynamics for modern Hopfield models, termed $\mathtt{U\text{-}Hop}$, with enhanced memory capacity. Our key contribution is a learnable feature map $\Phi$ which transforms the Hopfield energy function into kernel space. This transformation ensures convergence between the local minima of energy and the fixed points of retrieval dynamics within the kernel space. Consequently, the kernel norm induced by $\Phi$ serves as a novel similarity measure. It utilizes the stored memory patterns as learning data to enhance memory capacity across all modern Hopfield models. Specifically, we accomplish this by constructing a separation loss $\mathcal{L}_\Phi$ that separates the local minima of kernelized energy by separating stored memory patterns in kernel space. Methodologically, $\mathtt{U\text{-}Hop}$ memory retrieval process consists of: (Stage I) minimizing separation loss for a more uniform memory (local minimum) distribution, followed by (Stage II) standard Hopfield energy minimization for memory retrieval. This results in a significant reduction of possible metastable states in the Hopfield energy function, thus enhancing memory capacity by preventing memory confusion. Empirically, with real-world datasets, we demonstrate that $\mathtt{U\text{-}Hop}$ outperforms all existing modern Hopfield models and state-of-the-art similarity measures, achieving substantial improvements in both associative memory retrieval and deep learning tasks. Code is available at <a class="link-external link-https" href="https://github.com/MAGICS-LAB/UHop" rel="external noopener nofollow">this https URL</a> ; future updates are on <a class="link-https" data-arxiv-id="2404.03827" href="https://arxiv.org/abs/2404.03827">arXiv:2404.03827</a>

A generalized Hopfield model to store and retrieve mismatched memory patterns

Effect of spatial correlations on Hopfield Neural Network and Dense Associative Memories

High storage capacity in the Hopfield model with auto-interactions - stability analysis

Out-of-equilibrium dynamics of the Hopfield model in its spin-glass phase

Storage and Learning phase transitions in the Random-Features Hopfield Model

On a model of associative memory with huge storage capacity

Storage Capacity of the Hopfield Network Associative Memory

Retrieval dynamics of associative memory of the Hopfield type

Dreaming neural networks: Forgetting spurious memories and reinforcing pure ones

Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models

A Hopfield-like model with complementary encodings of memories

Non-Convex Multi-species Hopfield models

Input-Driven Dynamics for Robust Memory Retrieval in Hopfield Networks

Hopfield model with planted patterns: a teacher-student self-supervised learning model

Daydreaming Hopfield Networks and their surprising effectiveness on correlated data

The capacity of the Hopfield associative memory

Long Sequence Hopfield Memory

Capacity of the Hebbian-Hopfield network associative memory

Mean-field Monte Carlo Approach to the Dynamics of a One Pattern Model of Associative Memory

Pattern Storage and Hopfield Neural Associative Memory with Hidden Structure.

On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysis