Abstract:Restricted Boltzmann Machines (RBMs) are effective tools for modeling complex systems and deriving insights from data. However, training these models with highly structured data presents significant challenges due to the slow mixing characteristics of Markov Chain Monte Carlo processes. In this study, we build upon recent theoretical advancements in RBM training, to significantly reduce the computational cost of training (in very clustered datasets), evaluating and sampling in RBMs in general. The learning process is analogous to thermodynamic continuous phase transitions observed in ferromagnetic models, where new modes in the probability measure emerge in a continuous manner. Such continuous transitions are associated with the critical slowdown effect, which adversely affects the accuracy of gradient estimates, particularly during the initial stages of training with clustered data. To mitigate this issue, we propose a pre-training phase that encodes the principal components into a low-rank RBM through a convex optimization process. This approach enables efficient static Monte Carlo sampling and accurate computation of the partition function. We exploit the continuous and smooth nature of the parameter annealing trajectory to achieve reliable and computationally efficient log-likelihood estimations, enabling online assessment during the training, and propose a novel sampling strategy named parallel trajectory tempering (PTT) which outperforms previously optimized MCMC methods. Our results show that this training strategy enables RBMs to effectively address highly structured datasets that conventional methods struggle with. We also provide evidence that our log-likelihood estimation is more accurate than traditional, more computationally intensive approaches in controlled scenarios. The PTT algorithm significantly accelerates MCMC processes compared to existing and conventional methods.

Generalization properties of restricted Boltzmann machine for short-range order

Training Restricted Boltzmann Machines with Binary Synapses Using the Bayesian Learning Rule

Fast training and sampling of Restricted Boltzmann Machines

Nonequilibrium Thermodynamics of Restricted Boltzmann Machines

Thermodynamics of the Ising Model Encoded in Restricted Boltzmann Machines

Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit

Thermodynamics of Restricted Boltzmann Machines and related learning dynamics

Theory of binary alloys including short-range order properties

Identifying Product Order with Restricted Boltzmann Machines

Restricted Boltzmann Machine, recent advances and mean-field theory

Inferring effective couplings with Restricted Boltzmann Machines

Characterizing nanomagnetic arrays using restricted Boltzmann machines

Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function

Properties and Bayesian fitting of restricted Boltzmann machines

Restricted Boltzmann machine based algorithm for multi-objective optimization

On the equivalence of Hopfield networks and Boltzmann Machines

Short-sighted deep learning

Barriers and dynamical paths in alternating Gibbs sampling of restricted Boltzmann machines

Accuracy of restricted Boltzmann machines for the one-dimensional $J_1-J_2$ Heisenberg model

Learning thermodynamics and topological order of the 2D XY model with generative real-valued restricted Boltzmann machines

Configuration Interaction Guided Sampling with Interpretable Restricted Boltzmann Machine