Abstract:Restricted Boltzmann Machines (RBMs) are effective tools for modeling complex systems and deriving insights from data. However, training these models with highly structured data presents significant challenges due to the slow mixing characteristics of Markov Chain Monte Carlo processes. In this study, we build upon recent theoretical advancements in RBM training, to significantly reduce the computational cost of training (in very clustered datasets), evaluating and sampling in RBMs in general. The learning process is analogous to thermodynamic continuous phase transitions observed in ferromagnetic models, where new modes in the probability measure emerge in a continuous manner. Such continuous transitions are associated with the critical slowdown effect, which adversely affects the accuracy of gradient estimates, particularly during the initial stages of training with clustered data. To mitigate this issue, we propose a pre-training phase that encodes the principal components into a low-rank RBM through a convex optimization process. This approach enables efficient static Monte Carlo sampling and accurate computation of the partition function. We exploit the continuous and smooth nature of the parameter annealing trajectory to achieve reliable and computationally efficient log-likelihood estimations, enabling online assessment during the training, and propose a novel sampling strategy named parallel trajectory tempering (PTT) which outperforms previously optimized MCMC methods. Our results show that this training strategy enables RBMs to effectively address highly structured datasets that conventional methods struggle with. We also provide evidence that our log-likelihood estimation is more accurate than traditional, more computationally intensive approaches in controlled scenarios. The PTT algorithm significantly accelerates MCMC processes compared to existing and conventional methods.

Fuzzy Restricted Boltzmann Machine and Its Fast Learning Algorithm

A Fuzzy Restricted Boltzmann Machine: Novel Learning Algorithms Based on the Crisp Possibilistic Mean Value of Fuzzy Numbers

Infinite Fuzzy Restricted Boltzmann Machines for the Enhancement of Fuzzy Neural Networks

A Novel Restricted Boltzmann Machine Training Algorithm with Fast Gibbs Sampling Policy

Fuzzy Restricted Boltzmann Machine and Deep Belief Network: A Comparison on Image Reconstruction.

A Fuzzy Deep Model Based on Fuzzy Restricted Boltzmann Machines for High-dimensional Data Classification

Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy

Sparse Bayesian Approach to Fast Learning Network for Multiclassification.

An automatic setting for training restricted boltzmann machine

MapReduce Based Distributed Learning Algorithm for Restricted Boltzmann Machine

Restricted Boltzmann Machine with Adaptive Local Hidden Units.

Generative and Discriminative Infinite Restricted Boltzmann Machine Training

A Precise Method for RBMs Training Using Phased Curricula

Accelerated Learning for Restricted Boltzmann Machine with a Novel Momentum Algorithm

An Overview on Restricted Boltzmann Machines.

Data-Driven Fuzzy Modeling Using Restricted Boltzmann Machines and Probability Theory

Data-Driven Fuzzy Modeling Using Deep Learning

Real-Time Keypoint Recognition Using Restricted Boltzmann Machine

Generalized Boltzmann Machine with Deep Neural Structure.

Fast training and sampling of Restricted Boltzmann Machines

Deep Restricted Boltzmann Networks