Abstract:Restricted Boltzmann machines (RBMs) are commonly used as pre-training methods for deep learning models. Contrastive divergence (CD) and parallel tempering (PT) are traditional training algorithms of RBMs. However, these two algorithms have shortcomings in processing high-dimensional and complex data. In particular, the number of temperature chains in PT has a significant impact on the training effect, and the PT algorithm cannot fully utilize parallel sampling from multiple temperature chains for the divergence of the algorithm. The training can quickly converge with fewer temperature chains, but this impacts the accuracy. More temperature chains can help PT achieve higher accuracy in theory, but severe divergence at the beginning of the training may ruin the training result. To exploit fully the advantages of PT and improve the ability of RBMs to process high-dimensional and complex models, this article proposes dynamic tempering chains (DTC). By dynamically changing the number of temperature chains during the training process, DTC starts training with fewer temperature chains and gradually increase the number of temperature chains with training going on, and finally get an accurate RBM. And one-step reconstruction error is proposed to measure the convergence, which can decrease the influence of the dynamic training strategy on reconstruction error. Experiments on MNIST, MNORB, Cifar 10, and Cifar 100 indicate that, compared with PT, the classification accuracy of DTC algorithm improved by up to 8%. DTC quickly converges in the early stage of training because of few exchanges among temperature chains and produces higher accuracy at the end for the global optimum model learned by more temperature chains, especially when learning high-dimensional and complex data. This proves that the DTC algorithm effectively utilizes parallel sampling of multiple temperature chains, overcomes divergence challenges, and further improves the training effect of the RBM.

Rényi Divergence Based Generalization for Learning of Classification Restricted Boltzmann Machines

Training Restricted Boltzmann Machines with Binary Synapses Using the Bayesian Learning Rule

Generative and Discriminative Infinite Restricted Boltzmann Machine Training

Method to Improve the Performance of Restricted Boltzmann Machines.

Ratio Divergence Learning Using Target Energy in Restricted Boltzmann Machines: Beyond Kullback--Leibler Divergence Learning

A Novel Restricted Boltzmann Machine Training Algorithm with Dynamic Tempering Chains.

Classification Model of Restricted Boltzmann Machine Based on Reconstruction Error.

Learning Class-relevant Features and Class-irrelevant Features Via a Hybrid Third-Order RBM.

Average Contrastive Divergence for Training Restricted Boltzmann Machines.

A Precise Method for RBMs Training Using Phased Curricula

Generalization Improvement for Regularized Least Squares Classification

A Cyclic Contrastive Divergence Learning Algorithm for High-Order RBMs

Learning Discriminative Representation with Signed Laplacian Restricted Boltzmann Machine

Penalized Bregman divergence for large-dimensional regression and classification.

Contrastive Divergence Learning of Restricted Boltzmann Machine

An Overview on Restricted Boltzmann Machines.

Radial Basis Function Network Learning Using Localized Generalization Error Bound

Data normalization in the learning of restricted Boltzmann machines

Adversarial Training Methods for Boltzmann Machines

Discriminative Matrix-Variate Restricted Boltzmann Machine Classification Model

An automatic setting for training restricted boltzmann machine