Abstract:Restricted Boltzmann machines (RBMs) constitute one of the main models for machine statistical inference and they are widely employed in Artificial Intelligence as powerful tools for (deep) learning. However, in contrast with countless remarkable practical successes, their mathematical formalization has been largely elusive: from a statistical-mechanics perspective these systems display the same (random) Gibbs measure of bi-partite spin-glasses, whose rigorous treatment is notoriously difficult. In this work, beyond providing a brief review on RBMs from both the learning and the retrieval perspectives, we aim to contribute to their analytical investigation, by considering two distinct realizations of their weights (i.e., Boolean and Gaussian) and studying the properties of their related free energies. More precisely, focusing on a RBM characterized by digital couplings, we first extend the Pastur-Shcherbina-Tirozzi method (originally developed for the Hopfield model) to prove the self-averaging property for the free energy, over its quenched expectation, in the infinite volume limit, then we explicitly calculate its simplest approximation, namely its annealed bound. Next, focusing on a RBM characterized by analogical weights, we extend Guerra's interpolating scheme to obtain a control of the quenched free-energy under the assumption of replica symmetry: we get self-consistencies for the order parameters (in full agreement with the existing Literature) as well as the critical line for ergodicity breaking that turns out to be the same obtained in AGS theory. As we discuss, this analogy stems from the slow-noise universality. Finally, glancing beyond replica symmetry, we analyze the fluctuations of the overlaps for an estimate of the (slow) noise affecting the retrieval of the signal, and by a stability analysis we recover the Aizenman-Contucci identities typical of glassy systems.

On the equivalence of Hopfield networks and Boltzmann Machines

Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks

Training Restricted Boltzmann Machines with Binary Synapses Using the Bayesian Learning Rule

Nonequilibrium Thermodynamics of Restricted Boltzmann Machines

Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines

Riemann-Theta Boltzmann Machine

Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit

Thermodynamics of bidirectional associative memories

A new mechanical approach to handle generalized Hopfield neural networks

Inferring effective couplings with Restricted Boltzmann Machines

Thermodynamics of Restricted Boltzmann Machines and related learning dynamics

Accuracy of restricted Boltzmann machines for the one-dimensional $J_1-J_2$ Heisenberg model

Monotone deep Boltzmann machines

Thermodynamics of the Ising Model Encoded in Restricted Boltzmann Machines

Restricted Boltzmann Machine, recent advances and mean-field theory

Restricted Boltzmann Machines: Introduction and Review

The emergence of a concept in shallow neural networks

Attention in a Family of Boltzmann Machines Emerging From Modern Hopfield Networks

Short-sighted deep learning

An Overview of Restricted Boltzmann Machines