A new frontier for Hopfield networks

Dmitry Krotov

DOI: https://doi.org/10.1038/s42254-023-00595-y

IF: 36.273

2023-05-18

Nature Reviews Physics

Abstract:Over the past few years there has been a resurgence of interest in Hopfield networks of associative memory. Dmitry Krotov discusses recent theoretical advances and their broader impact in the context of energy-based neural architectures.

physics, applied, multidisciplinary

What problem does this paper attempt to address?

Nobel honors for John Hopfield, who ushered attractor dynamics into neuroscience

David Kleinfeld

DOI: https://doi.org/10.1016/j.neuron.2024.11.002

IF: 16.2

2024-12-07

Neuron

Abstract:John Hopfield's model on collective computation linked the recall of memories with interactions and dynamics associated with disordered magnetic systems. Insights from Hopfield's work catalyzed formulations that link the dynamics and emergent properties of recurrently connected generic neurons with the functional properties and signaling observed from brain circuits.

neurosciences
Sparse and Structured Hopfield Networks

Saul Santos,Vlad Niculae,Daniel McNamee,Andre F. T. Martins

2024-06-05

Abstract:Modern Hopfield networks have enjoyed recent interest due to their connection to attention in transformers. Our paper provides a unified framework for sparse Hopfield networks by establishing a link with Fenchel-Young losses. The result is a new family of Hopfield-Fenchel-Young energies whose update rules are end-to-end differentiable sparse transformations. We reveal a connection between loss margins, sparsity, and exact memory retrieval. We further extend this framework to structured Hopfield networks via the SparseMAP transformation, which can retrieve pattern associations instead of a single pattern. Experiments on multiple instance learning and text rationalization demonstrate the usefulness of our approach.

Machine Learning
Input-Driven Dynamics for Robust Memory Retrieval in Hopfield Networks

Simone Betteti,Giacomo Baggio,Francesco Bullo,Sandro Zampieri

2024-11-07

Abstract:The Hopfield model provides a mathematically idealized yet insightful framework for understanding the mechanisms of memory storage and retrieval in the human brain. This model has inspired four decades of extensive research on learning and retrieval dynamics, capacity estimates, and sequential transitions among memories. Notably, the role and impact of external inputs has been largely underexplored, from their effects on neural dynamics to how they facilitate effective memory retrieval. To bridge this gap, we propose a novel dynamical system framework in which the external input directly influences the neural synapses and shapes the energy landscape of the Hopfield model. This plasticity-based mechanism provides a clear energetic interpretation of the memory retrieval process and proves effective at correctly classifying highly mixed inputs. Furthermore, we integrate this model within the framework of modern Hopfield architectures, using this connection to elucidate how current and past information are combined during the retrieval process. Finally, we embed both the classic and the new model in an environment disrupted by noise and compare their robustness during memory retrieval.

Neurons and Cognition,Statistical Mechanics,Artificial Intelligence,Machine Learning,Dynamical Systems
Simplicial Hopfield networks

Thomas F Burns,Tomoki Fukai

2023-05-09

Abstract:Hopfield networks are artificial neural networks which store memory patterns on the states of their neurons by choosing recurrent connection weights and update rules such that the energy landscape of the network forms attractors around the memories. How many stable, sufficiently-attracting memory patterns can we store in such a network using $N$ neurons? The answer depends on the choice of weights and update rule. Inspired by setwise connectivity in biology, we extend Hopfield networks by adding setwise connections and embedding these connections in a simplicial complex. Simplicial complexes are higher dimensional analogues of graphs which naturally represent collections of pairwise and setwise relationships. We show that our simplicial Hopfield networks increase memory storage capacity. Surprisingly, even when connections are limited to a small random subset of equivalent size to an all-pairwise network, our networks still outperform their pairwise counterparts. Such scenarios include non-trivial simplicial topology. We also test analogous modern continuous Hopfield networks, offering a potentially promising avenue for improving the attention mechanism in Transformer models.

Neural and Evolutionary Computing,Artificial Intelligence,Neurons and Cognition
Large Associative Memory Problem in Neurobiology and Machine Learning

Dmitry Krotov,John Hopfield

DOI: https://doi.org/10.48550/arXiv.2008.06996

2021-04-28

Abstract:Dense Associative Memories or modern Hopfield networks permit storage and reliable retrieval of an exponentially large (in the dimension of feature space) number of memories. At the same time, their naive implementation is non-biological, since it seemingly requires the existence of many-body synaptic junctions between the neurons. We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them. For this reason our proposed microscopic theory is a valid model of large associative memory with a degree of biological plausibility. The dynamics of our network and its reduced dimensional equivalent both minimize energy (Lyapunov) functions. When certain dynamical variables (hidden neurons) are integrated out from our microscopic theory, one can recover many of the models that were previously discussed in the literature, e.g. the model presented in "Hopfield Networks is All You Need" paper. We also provide an alternative derivation of the energy function and the update rule proposed in the aforementioned paper and clarify the relationships between various models of this class.

Neurons and Cognition,Disordered Systems and Neural Networks,Computation and Language,Machine Learning
Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models

Beren Millidge,Tommaso Salvatori,Yuhang Song,Thomas Lukasiewicz,Rafal Bogacz

Abstract:A large number of neural network models of associative memory have been proposed in the literature. These include the classical Hopfield networks (HNs), sparse distributed memories (SDMs), and more recently the modern continuous Hopfield networks (MCHNs), which possess close links with self-attention in machine learning. In this paper, we propose a general framework for understanding the operation of such memory networks as a sequence of three operations: similarity, separation, and projection. We derive all these memory models as instances of our general framework with differing similarity and separation functions. We extend the mathematical framework of Krotov & Hopfield (2020) to express general associative memory models using neural network dynamics with local computation, and derive a general energy function that is a Lyapunov function of the dynamics. Finally, using our framework, we empirically investigate the capacity of using different similarity functions for these associative memory models, beyond the dot product similarity measure, and demonstrate empirically that Euclidean or Manhattan distance similarity metrics perform substantially better in practice on many tasks, enabling a more robust retrieval and higher memory capacity than existing models.
Accelerating Hopfield Network Dynamics: Beyond Synchronous Updates and Forward Euler

Cédric Goemaere,Johannes Deleu,Thomas Demeester

2024-08-22

Abstract:The Hopfield network serves as a fundamental energy-based model in machine learning, capturing memory retrieval dynamics through an ordinary differential equation (ODE). The model's output, the equilibrium point of the ODE, is traditionally computed via synchronous updates using the forward Euler method. This paper aims to overcome some of the disadvantages of this approach. We propose a conceptual shift, viewing Hopfield networks as instances of Deep Equilibrium Models (DEQs). The DEQ framework not only allows for the use of specialized solvers, but also leads to new insights on an empirical inference technique that we will refer to as 'even-odd splitting'. Our theoretical analysis of the method uncovers a parallelizable asynchronous update scheme, which should converge roughly twice as fast as the conventional synchronous updates. Empirical evaluations validate these findings, showcasing the advantages of both the DEQ framework and even-odd splitting in digitally simulating energy minimization in Hopfield networks. The code is available at <a class="link-external link-https" href="https://github.com/cgoemaere/hopdeq" rel="external noopener nofollow">this https URL</a>

Machine Learning,Neural and Evolutionary Computing
Analysis of Hopfield Model as Associative Memory

Matteo Silvestri

2024-01-25

Abstract:This article delves into the Hopfield neural network model, drawing inspiration from biological neural systems. The exploration begins with an overview of the model's foundations, incorporating insights from mechanical statistics to deepen our understanding. Focusing on audio retrieval, the study demonstrates the Hopfield model's associative memory capabilities. Through practical implementation, the network is trained to retrieve different patterns.

Disordered Systems and Neural Networks
Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval

Saul Santos,Vlad Niculae,Daniel McNamee,André F. T. Martins

2024-11-13

Abstract:Associative memory models, such as Hopfield networks and their modern variants, have garnered renewed interest due to advancements in memory capacity and connections with self-attention in transformers. In this work, we introduce a unified framework-Hopfield-Fenchel-Young networks-which generalizes these models to a broader family of energy functions. Our energies are formulated as the difference between two Fenchel-Young losses: one, parameterized by a generalized entropy, defines the Hopfield scoring mechanism, while the other applies a post-transformation to the Hopfield output. By utilizing Tsallis and norm entropies, we derive end-to-end differentiable update rules that enable sparse transformations, uncovering new connections between loss margins, sparsity, and exact retrieval of single memory patterns. We further extend this framework to structured Hopfield networks using the SparseMAP transformation, allowing the retrieval of pattern associations rather than a single pattern. Our framework unifies and extends traditional and modern Hopfield networks and provides an energy minimization perspective for widely used post-transformations like $\ell_2$-normalization and layer normalization-all through suitable choices of Fenchel-Young losses and by using convex analysis as a building block. Finally, we validate our Hopfield-Fenchel-Young networks on diverse memory recall tasks, including free and sequential recall. Experiments on simulated data, image retrieval, multiple instance learning, and text rationalization demonstrate the effectiveness of our approach.

Machine Learning
Nonparametric Modern Hopfield Models

Jerry Yao-Chieh Hu,Bo-Yu Chen,Dennis Wu,Feng Ruan,Han Liu

2024-04-05

Abstract:We present a nonparametric construction for deep learning compatible modern Hopfield models and utilize this framework to debut an efficient variant. Our key contribution stems from interpreting the memory storage and retrieval processes in modern Hopfield models as a nonparametric regression problem subject to a set of query-memory pairs. Crucially, our framework not only recovers the known results from the original dense modern Hopfield model but also fills the void in the literature regarding efficient modern Hopfield models, by introducing \textit{sparse-structured} modern Hopfield models with sub-quadratic complexity. We establish that this sparse model inherits the appealing theoretical properties of its dense analogue -- connection with transformer attention, fixed point convergence and exponential memory capacity -- even without knowing details of the Hopfield energy function. Additionally, we showcase the versatility of our framework by constructing a family of modern Hopfield models as extensions, including linear, random masked, top-$K$ and positive random feature modern Hopfield models. Empirically, we validate the efficacy of our framework in both synthetic and realistic settings.

Machine Learning,Artificial Intelligence,Neural and Evolutionary Computing
Analysis of Discrete Modern Hopfield Networks in Open Quantum System

Takeshi Kimura,Kohtaro Kato

2024-11-05

Abstract:The modern Hopfield network, proposed by Krotov and Hopfield, is a mathematical generalization of the Hopfield network, which is a basic model of associative memory that employs higher-order interactions. This study introduces an open quantum model for discrete modern Hopfield networks that generalizes the open quantum Hopfield network. Our model integrates dissipative quantum spin systems, governed by quantum master equations, with classical hopping terms and additional quantum effects through a transverse field. We analytically examined the behavior of the stable fixed points and numerically determined the phase diagram. The results demonstrated qualitatively distinct behaviors from the open quantum Hopfield network, showing that the ferromagnetic and limit cycle phases have additional stable fixed points.

Quantum Physics,Disordered Systems and Neural Networks,Statistical Mechanics
Estimation of domain of attraction and exponential convergence rate of Hopfield's continuous-time associative memory and its applications

Xuebin Liang,Lide Wu

1996-01-01

Tien Tzu Hsueh Pao/Acta Electronica Sinica

Abstract:Some results about estimation of the domain of attraction of memory patterns and exponential convergence rate of the network trajectories to memory patterns for Hopfield's continuous-time associative memory were obtained. These results can be used not only for evaluation of error-correction capability of Hopfield continuous-time associative memories, but also for the synthesis of efficient continuous-time associative memory neural networks.
Temporal Complexity of a Hopfield-Type Neural Model in Random and Scale-Free Graphs

Marco Cafiso,Paolo Paradisi

2024-07-19

Abstract:The Hopfield network model and its generalizations were introduced as a model of associative, or content-addressable, memory. They were widely investigated both as a unsupervised learning method in artificial intelligence and as a model of biological neural dynamics in computational neuroscience. The complexity features of biological neural networks are attracting the interest of scientific community since the last two decades. More recently, concepts and tools borrowed from complex network theory were applied to artificial neural networks and learning, thus focusing on the topological aspects. However, the temporal structure is also a crucial property displayed by biological neural networks and investigated in the framework of systems displaying complex intermittency. The Intermittency-Driven Complexity (IDC) approach indeed focuses on the metastability of self-organized states, whose signature is a power-decay in the inter-event time distribution or a scaling behavior in the related event-driven diffusion processes. The investigation of IDC in neural dynamics and its relationship with network topology is still in its early stages. In this work we present the preliminary results of a IDC analysis carried out on a bio-inspired Hopfield-type neural network comparing two different connectivities, i.e., scale-free vs. random network topology. We found that random networks can trigger complexity features similar to that of scale-free networks, even if with some differences and for different parameter values, in particular for different noise levels.

Neurons and Cognition,Disordered Systems and Neural Networks,Mathematical Physics,Numerical Analysis,Adaptation and Self-Organizing Systems
Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks

Chiara Marullo,Elena Agliari

DOI: https://doi.org/10.3390/e23010034

IF: 2.738

2020-12-29

Entropy

Abstract:The Hopfield model and the Boltzmann machine are among the most popular examples of neural networks. The latter, widely used for classification and feature detection, is able to efficiently learn a generative model from observed data and constitutes the benchmark for statistical learning. The former, designed to mimic the retrieval phase of an artificial associative memory lays in between two paradigmatic statistical mechanics models, namely the Curie-Weiss and the Sherrington-Kirkpatrick, which are recovered as the limiting cases of, respectively, one and many stored memories. Interestingly, the Boltzmann machine and the Hopfield network, if considered to be two cognitive processes (learning and information retrieval), are nothing more than two sides of the same coin. In fact, it is possible to exactly map the one into the other. We will inspect such an equivalence retracing the most representative steps of the research in this field.

physics, multidisciplinary
On the Expressive Power of Modern Hopfield Networks

Xiaoyu Li,Yuanpeng Li,Yingyu Liang,Zhenmei Shi,Zhao Song

2024-12-07

Abstract:Modern Hopfield networks (MHNs) have emerged as powerful tools in deep learning, capable of replacing components such as pooling layers, LSTMs, and attention mechanisms. Recent advancements have enhanced their storage capacity, retrieval speed, and error rates. However, the fundamental limits of their computational expressiveness remain unexplored. Understanding the expressive power of MHNs is crucial for optimizing their integration into deep learning architectures. In this work, we establish rigorous theoretical bounds on the computational capabilities of MHNs using circuit complexity theory. Our key contribution is that we show that MHNs are $\mathsf{DLOGTIME}$-uniform $\mathsf{TC}^0$. Hence, unless $\mathsf{TC}^0 = \mathsf{NC}^1$, a $\mathrm{poly}(n)$-precision modern Hopfield networks with a constant number of layers and $O(n)$ hidden dimension cannot solve $\mathsf{NC}^1$-hard problems such as the undirected graph connectivity problem and the tree isomorphism problem. We also extended our results to Kernelized Hopfield Networks. These results demonstrate the limitation in the expressive power of the modern Hopfield networks. Moreover, Our theoretical analysis provides insights to guide the development of new Hopfield-based architectures.

Computational Complexity,Artificial Intelligence,Computation and Language,Machine Learning
Reconstructing creative thoughts: Hopfield neural networks

Denisa Checiu,Mathias Bode,Radwa Khalil

DOI: https://doi.org/10.1016/j.neucom.2024.127324

IF: 6

2024-01-28

Neurocomputing

Abstract:From a brain processing perspective, the perception of creative thinking is rooted in the underlying cognitive process, which facilitates exploring and cultivating novel avenues and problem-solving strategies. However, it is challenging to emulate the intricate complexity of how the human brain presents a novel way to uncover unique solutions. One potential approach to mitigating this complexity is incorporating creative cognition into the evolving artificial intelligence systems and associated neural models. Hopfield neural networks (HNN) are commonly acknowledged as a simplified neural model, renowned for their biological plausibility to store and retrieve information, specifically patterns of neurons. Our findings suggest utilizing modern HNN to emulate creative thinking by making meaningful associations between seemingly disparate concepts. This semantic link is represented as a radio knob that can be set to determine whether the network solves problems creatively or shuts down; the threshold is a parameter. We used the term "first knob of creativity" to describe a certain pattern and utilized the "second knob of creativity" to aid in the examination of alternatives within the network. By manipulating the knobs, it is possible to selectively suppress specific patterns, facilitating the creative functioning of the HNN and identifying other patterns with which input can be linked.

computer science, artificial intelligence
A New Kind of Hopfield Networks for Finding Global Optimum

Xiaofei Huang

DOI: https://doi.org/10.48550/arXiv.cs/0505003

2005-04-30

Neural and Evolutionary Computing

Abstract:The Hopfield network has been applied to solve optimization problems over decades. However, it still has many limitations in accomplishing this task. Most of them are inherited from the optimization algorithms it implements. The computation of a Hopfield network, defined by a set of difference equations, can easily be trapped into one local optimum or another, sensitive to initial conditions, perturbations, and neuron update orders. It doesn't know how long it will take to converge, as well as if the final solution is a global optimum, or not. In this paper, we present a Hopfield network with a new set of difference equations to fix those problems. The difference equations directly implement a new powerful optimization algorithm.
A Feed-Forward Neural Network for Increasing the Hopfield-Network Storage Capacity

Shaokai Zhao,Bin Chen,Hui Wang,Zhiyuan Luo,Tao Zhang

DOI: https://doi.org/10.1142/S0129065722500277

IF: 6.325

2022-05-11

International Journal of Neural Systems

Abstract:In the hippocampal dentate gyrus (DG), pattern separation mainly depends on the concepts of 'expansion recoding', meaning random mixing of different DG input channels. However, recent advances in neurophysiology have challenged the theory of pattern separation based on these concepts. In this study, we propose a novel feed-forward neural network, inspired by the structure of the DG and neural oscillatory analysis, to increase the Hopfield-network storage capacity. Unlike the previously published feed-forward neural networks, our bio-inspired neural network is designed to take advantage of both biological structure and functions of the DG. To better understand the computational principles of pattern separation in the DG, we have established a mouse model of environmental enrichment. We obtained a possible computational model of the DG, associated with better pattern separation ability, by using neural oscillatory analysis. Furthermore, we have developed a new algorithm based on Hebbian learning and coupling direction of neural oscillation to train the proposed neural network. The simulation results show that our proposed network significantly expands the storage capacity of Hopfield network, and more effective pattern separation is achieved. The storage capacity rises from 0.13 for the standard Hopfield network to 0.32 using our model when the overlap in patterns is 10%.

computer science, artificial intelligence
Brief Communication A Theorem Concerning the Energy Function of Hopfield Continuous-Variable Neural Networks

LUO FA-LONG,LI YAN-DA

DOI: https://doi.org/10.1080/00207219408925941

1994-01-01

International Journal of Electronics

Abstract:A theorem is presented concerning the energy function of Hopfield continuous-variable neural networks, which is useful for designing the circuit implementation of such neural networks and analysing their performance.

A new frontier for Hopfield networks

Nobel honors for John Hopfield, who ushered attractor dynamics into neuroscience

Sparse and Structured Hopfield Networks

Input-Driven Dynamics for Robust Memory Retrieval in Hopfield Networks

Simplicial Hopfield networks

Large Associative Memory Problem in Neurobiology and Machine Learning

Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models

Accelerating Hopfield Network Dynamics: Beyond Synchronous Updates and Forward Euler

Analysis of Hopfield Model as Associative Memory

Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval

Nonparametric Modern Hopfield Models

Analysis of Discrete Modern Hopfield Networks in Open Quantum System

Estimation of domain of attraction and exponential convergence rate of Hopfield's continuous-time associative memory and its applications

Temporal Complexity of a Hopfield-Type Neural Model in Random and Scale-Free Graphs

Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks

On the Expressive Power of Modern Hopfield Networks

Reconstructing creative thoughts: Hopfield neural networks

A New Kind of Hopfield Networks for Finding Global Optimum

A Feed-Forward Neural Network for Increasing the Hopfield-Network Storage Capacity

Brief Communication A Theorem Concerning the Energy Function of Hopfield Continuous-Variable Neural Networks