Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
Aurélien Decelle,Giancarlo Fissore,Cyril Furtlehner
DOI: https://doi.org/10.1007/s10955-018-2105-y
2018-03-06
Abstract:We investigate the thermodynamic properties of a Restricted Boltzmann Machine
(RBM), a simple energy-based generative model used in the context of
unsupervised learning. Assuming the information content of this model to be
mainly reflected by the spectral properties of its weight matrix $W$, we try to
make a realistic analysis by averaging over an appropriate statistical ensemble
of RBMs.
First, a phase diagram is derived. Otherwise similar to that of the
Sherrington- Kirkpatrick (SK) model with ferromagnetic couplings, the RBM's
phase diagram presents a ferromagnetic phase which may or may not be of
compositional type depending on the kurtosis of the distribution of the
components of the singular vectors of $W$.
Subsequently, the learning dynamics of the RBM is studied in the
thermodynamic limit. A "typical" learning trajectory is shown to solve an
effective dynamical equation, based on the aforementioned ensemble average and
explicitly involving order parameters obtained from the thermodynamic analysis.
In particular, this let us show how the evolution of the dominant singular
values of $W$, and thus of the unstable modes, is driven by the input data. At
the beginning of the training, in which the RBM is found to operate in the
linear regime, the unstable modes reflect the dominant covariance modes of the
data. In the non-linear regime, instead, the selected modes interact and
eventually impose a matching of the order parameters to their empirical
counterparts estimated from the data.
Finally, we illustrate our considerations by performing experiments on both
artificial and real data, showing in particular how the RBM operates in the
ferromagnetic compositional phase.
Disordered Systems and Neural Networks,Machine Learning,Statistical Mechanics