Abstract:A specific type of neural networks, the Restricted Boltzmann Machines (RBM), are implemented for classification and feature detection in machine learning. They are characterized by separate layers of visible and hidden units, which are able to learn efficiently a generative model of the observed data. We study a "hybrid" version of RBMs, in which hidden units are analog and visible units are binary, and we show that thermodynamics of visible units are equivalent to those of a Hopfield network, in which the N visible units are the neurons and the P hidden units are the learned patterns. We apply the method of stochastic stability to derive the thermodynamics of the model, by considering a formal extension of this technique to the case of multiple sets of stored patterns, which may act as a benchmark for the study of correlated sets. Our results imply that simulating the dynamics of a Hopfield network, requiring the update of N neurons and the storage of N(N-1)/2 synapses, can be accomplished by a hybrid Boltzmann Machine, requiring the update of N+P neurons but the storage of only NP synapses. In addition, the well known glass transition of the Hopfield network has a counterpart in the Boltzmann Machine: it corresponds to an optimum criterion for selecting the relative sizes of the hidden and visible layers, resolving the trade-off between flexibility and generality of the model. The low storage phase of the Hopfield model corresponds to few hidden units and hence a overly constrained RBM, while the spin-glass phase (too many hidden units) corresponds to unconstrained RBM prone to overfitting of the observed data.

Deep Narrow Boltzmann Machines are Universal Approximators

Deep, narrow sigmoid belief networks are universal approximators

Monotone deep Boltzmann machines

Restricted Boltzmann Machines: Introduction and Review

Universal Approximation of Markov Kernels by Shallow Stochastic Feedforward Networks

Training Restricted Boltzmann Machines with Binary Synapses Using the Bayesian Learning Rule

Universal representation by Boltzmann machines with Regularised Axons

Universal approximation with complex-valued deep narrow neural networks

Geometry and Expressive Power of Conditional Restricted Boltzmann Machines

Constructive Universal High-Dimensional Distribution Generation through Deep ReLU Networks

Universality of deep convolutional neural networks

Large Margin Boltzmann Machines and Large Margin Sigmoid Belief Networks

New advances in universal approximation with neural networks of minimal width

On Training Deep Boltzmann Machines

On the capacity of deep generative networks for approximating distributions

Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks

Riemann-Theta Boltzmann Machine

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

On the equivalence of Hopfield networks and Boltzmann Machines

Universal Approximation Capability of Broad Learning System and Its Structural Variations

A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit