In search of dispersed memories: Generative diffusion models are associative memory networks

Luca Ambrogioni

2023-11-18

Abstract:Uncovering the mechanisms behind long-term memory is one of the most fascinating open problems in neuroscience and artificial intelligence. Artificial associative memory networks have been used to formalize important aspects of biological memory. Generative diffusion models are a type of generative machine learning techniques that have shown great performance in many tasks. Like associative memory systems, these networks define a dynamical system that converges to a set of target states. In this work we show that generative diffusion models can be interpreted as energy-based models and that, when trained on discrete patterns, their energy function is (asymptotically) identical to that of modern Hopfield networks. This equivalence allows us to interpret the supervised training of diffusion models as a synaptic learning process that encodes the associative dynamics of a modern Hopfield network in the weight structure of a deep neural network. Leveraging this connection, we formulate a generalized framework for understanding the formation of long-term memory, where creative generation and memory recall can be seen as parts of a unified continuum.

Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the understanding of long-term memory mechanisms in the fields of neuroscience and artificial intelligence. Specifically, the authors try to associate generative diffusion models with modern Hopfield networks to establish a unified theoretical framework for understanding the formation of long-term memory. The main contributions of the paper are:

1. **Theoretical Equivalence**: The paper demonstrates that generative diffusion models can be interpreted as energy-based models, and when these models are trained on discrete patterns, their energy functions asymptotically match those of modern Hopfield networks. This equivalence allows the supervised training of generative diffusion models to be viewed as a process of encoding associative dynamics.

2. **Unified Framework**: Through this connection, the authors propose a generalized framework that views creative generation and memory recall as parts of the same continuum. This means that generative diffusion models can not only generate new samples but also be used for memory retrieval and reconstruction.

3. **Biological Relevance**: The paper further explores how this theoretical framework relates to memory mechanisms in biological systems, particularly the memory consolidation process associated with the hippocampus. The authors propose a hypothesis that new simulated experiences may be generated in hippocampal networks and used for synaptic training, thereby explaining the formation of long-term memory in humans and animals.

In summary, the paper aims to provide new perspectives and tools for understanding and simulating long-term memory mechanisms in biological systems through the theoretical bridge between generative diffusion models and modern Hopfield networks.