Abstract:Recently, the original storage prescription for the Hopfield model of neural networks -- as well as for its dense generalizations -- has been turned into a genuine Hebbian learning rule by postulating the expression of its Hamiltonian for both the supervised and unsupervised protocols. In these notes, first, we obtain these explicit expressions by relying upon maximum entropy extremization à la Jaynes. Beyond providing a formal derivation of these recipes for Hebbian learning, this construction also highlights how Lagrangian constraints within entropy extremization force network's outcomes on neural correlations: these try to mimic the empirical counterparts hidden in the datasets provided to the network for its training and, the denser the network, the longer the correlations that it is able to capture. Next, we prove that, in the big data limit, whatever the presence of a teacher (or its lacking), not only these Hebbian learning rules converge to the original storage prescription of the Hopfield model but also their related free energies (and, thus, the statistical mechanical picture provided by Amit, Gutfreund and Sompolinsky is fully recovered). As a sideline, we show mathematical equivalence among standard Cost functions (Hamiltonian), preferred in Statistical Mechanical jargon, and quadratic Loss Functions, preferred in Machine Learning terminology. Remarks on the exponential Hopfield model (as the limit of dense networks with diverging density) and semi-supervised protocols are also provided.

Some Exact Results of Hopfield Neural Networks and Applications

Estimation of domain of attraction and exponential convergence rate of Hopfield's continuous-time associative memory and its applications

A New Result on the Global Convergence of Hopfield Neural Networks

Effects of Delays on the Basin Boundary of Attraction in A Hopfield Network of Two Delay-Connecting Neurons

Stability Conditions for Discrete Hopfield Neural Networks with Delay

Effect of spatial correlations on Hopfield Neural Network and Dense Associative Memories

Stability Analysis for Delayed Hopfield Neural Networks

A New Kind of Hopfield Networks for Finding Global Optimum

Existence and Global Exponential Stability of Almost Automorphic Solution for Clifford-Valued High-Order Hopfield Neural Networks with Leakage Delays

Simplicial Hopfield networks

Global Exponential Stability in Hopfield and Bidirectional Associative Memory Neural Networks with Time Delays

The Exponential Asymptotic Stability For Hopfield Neural Networks With Delays And Impulses

Exponential Stability of Periodic Solution for Delayed Hopfield Networks

Brief Communication A Theorem Concerning the Energy Function of Hopfield Continuous-Variable Neural Networks

Storage Capacity of the Hopfield Network Associative Memory

Stability Of Iminulsive Delayed Hopfield Neural Networks

Statistical Mechanics of a Multiconnected Hopfield Neural-Network Model in a Transverse Field

Analysis of Hopfield Model as Associative Memory

Heteroclinic cycles in Hopfield networks

Hebbian Learning from First Principles

Statistical Mechanics of a Hopfield Neural-Network Model in a Transverse Field