Deep Restricted Boltzmann Networks

Hengyuan Hu,Lisheng Gao,Quanbin Ma

DOI: https://doi.org/10.48550/arXiv.1611.07917

2016-11-15

Abstract:Building a good generative model for image has long been an important topic in computer vision and machine learning. Restricted Boltzmann machine (RBM) is one of such models that is simple but powerful. However, its restricted form also has placed heavy constraints on the models representation power and scalability. Many extensions have been invented based on RBM in order to produce deeper architectures with greater power. The most famous ones among them are deep belief network, which stacks multiple layer-wise pretrained RBMs to form a hybrid model, and deep Boltzmann machine, which allows connections between hidden units to form a multi-layer structure. In this paper, we present a new method to compose RBMs to form a multi-layer network style architecture and a training method that trains all layers jointly. We call the resulted structure deep restricted Boltzmann network. We further explore the combination of convolutional RBM with the normal fully connected RBM, which is made trivial under our composition framework. Experiments show that our model can generate descent images and outperform the normal RBM significantly in terms of image quality and feature quality, without losing much efficiency for training.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the performance of the Restricted Boltzmann Machine (RBM) in image generation and feature extraction. Specifically, the traditional single - layer RBM has limitations in modeling complex probability distributions, especially when dealing with high - dimensional data such as images. To solve this problem, the author proposes a new method to combine RBMs to form a multi - layer network structure and develops a training method that can jointly train all layers. This new structure is called the Deep Restricted Boltzmann Network (DRBN). In addition, the author also explores the possibility of combining the convolutional RBM with the ordinary fully - connected RBM to improve the scalability and performance of the model.

The main contributions of the paper include:
1. **Proposing the DRBN architecture**: By stacking multiple RBMs to form a multi - layer network, each RBM is trained as an independent layer, but the entire network can be jointly optimized.
2. **Joint training method**: A method that can train all layers simultaneously has been developed, using Persistent Contrastive Divergence (PCD) to approximate model - dependent terms, thereby achieving effective parameter updates.
3. **Integration of convolutional RBM**: Combining the convolutional RBM with the ordinary RBM to form a more powerful model that can handle high - dimensional data such as images.
4. **Experimental verification**: Through experiments on the MNIST and Weizmann Horses datasets, the superior performance of DRBN in image generation and feature extraction tasks has been proven.

These improvements make DRBN significantly superior to the traditional single - layer RBM in image generation quality and also show better performance in feature extraction tasks, especially when labeled data is scarce.