Abstract:Statistical mechanics has made significant contributions to the study of biological neural systems by modeling them as recurrent networks of interconnected units with adjustable interactions. Several algorithms have been proposed to optimize the neural connections to enable network tasks such as information storage (i.e. associative memory) and learning probability distributions from data (i.e. generative modeling). Among these methods, the Unlearning algorithm, aligned with emerging theories of synaptic plasticity, was introduced by John Hopfield and collaborators. The primary objective of this thesis is to understand the effectiveness of Unlearning in both associative memory models and generative models. Initially, we demonstrate that the Unlearning algorithm can be simplified to a linear perceptron model which learns from noisy examples featuring specific internal correlations. The selection of structured training data enables an associative memory model to retrieve concepts as attractors of a neural dynamics with considerable basins of attraction. Subsequently, a novel regularization technique for Boltzmann Machines is presented, proving to outperform previously developed methods in learning hidden probability distributions from data-sets. The Unlearning rule is derived from this new regularized algorithm and is showed to be comparable, in terms of inferential performance, to traditional Boltzmann-Machine learning.

What problem does this paper attempt to address?

The paper primarily explores the issues of memory unlearning and reinforcement in spin-glass-like neural networks. Its core objective is to understand the effectiveness of the "Unlearning" algorithm in associative memory models and generative models. Specifically, the paper first demonstrates that the Unlearning algorithm can be simplified to a linear perceptron model and can learn from noise samples with specific internal correlations. By selecting structured training data, the associative memory model can retrieve concepts as attractors of neural dynamics, and these attractors have fairly large basins of attraction. Subsequently, the paper proposes a new regularization technique for Boltzmann machines and shows that this method outperforms previously developed methods in learning implicit probability distributions from datasets. The Unlearning rule is derived from this new regularization algorithm and is comparable to traditional Boltzmann machine learning methods in terms of inference performance. Overall, the work in this paper can be divided into three parts: 1. Demonstrating that the Unlearning rule naturally arises from a noise-injected perceptron algorithm and achieves optimal associative memory performance. Unlearning is used as a valuable unsupervised alternative method for maximizing the training of stable perceptrons (i.e., support vector machines). 2. Proposing a new regularization method for Boltzmann machines and investigating the inference capabilities of the standard Unlearning process as well as the importance of Boltzmann machine parameter initialization. 3. Utilizing the Unlearning algorithm as a bridge connecting two artificial intelligence learning frameworks: associative memory (and its formal mapping to classification problems) and generative modeling, particularly Boltzmann machine learning. Specifically, the paper attempts to demonstrate the formal equivalence among three learning algorithms: Unlearning, support vector machines, and Boltzmann machines. This paper links the knowledge of statistical mechanics, theoretical neuroscience, and artificial intelligence, providing useful insights for their unification.

Demolition and Reinforcement of Memories in Spin-Glass-like Neural Networks