Abstract:Restricted Boltzmann Machines are generative models that consist of a layer of hidden variables connected to another layer of visible units, and they are used to model the distribution over visible variables. In order to gain a higher representability power, many hidden units are commonly used, which, in combination with a large number of visible units, leads to a high number of trainable parameters. In this work we introduce the Structural Restricted Boltzmann Machine model, which taking advantage of the structure of the data in hand, constrains connections of hidden units to subsets of visible units in order to reduce significantly the number of trainable parameters, without compromising performance. As a possible area of application, we focus on image modelling. Based on the nature of the images, the structure of the connections is given in terms of spatial neighbourhoods over the pixels of the image that constitute the visible variables of the model. We conduct extensive experiments on various image domains. Image denoising is evaluated with corrupted images from the MNIST dataset. The generative power of our models is compared to vanilla RBMs, as well as their classification performance, which is assessed with five different image domains. Results show that our proposed model has a faster and more stable training, while also obtaining better results compared to an RBM with no constrained connections between its visible and hidden units.
What problem does this paper attempt to address?
This paper attempts to solve the problems of high training complexity and slow training speed caused by too many parameters when the traditional Restricted Boltzmann Machine (RBM) processes high - dimensional data by introducing the Structural Restricted Boltzmann Machine (SBM). Specifically, the main objectives of the paper are:
1. **Reduce the number of trainable parameters**: By restricting the connections between hidden units and visible units, the SBM model can significantly reduce the number of trainable parameters, thereby reducing the computational cost and improving the training efficiency.
2. **Maintain or improve performance**: Despite the reduction in the number of parameters, the SBM model can still maintain or even exceed the performance of the traditional RBM in tasks such as image denoising and classification.
3. **Utilize data structure**: The SBM model is especially suitable for data with a specific structure, such as images. By utilizing the spatial neighborhood relationship of pixels, the SBM can model image data more effectively.
### Paper background
- **Restricted Boltzmann Machine (RBM)**: RBM is a generative model composed of a layer of hidden units and a layer of visible units, which is used to model the distribution of visible variables. The traditional RBM has a large number of parameters due to the need for a large number of hidden units and visible units, resulting in a complex and time - consuming training process.
- **Problem**: In order to improve the representational ability of RBM, a large number of hidden units are usually used, which will lead to a surge in the number of parameters when processing large - scale data, increasing the computational cost and training time.
### Paper contributions
1. **Propose the Structural Restricted Boltzmann Machine (SBM)**: By restricting each hidden unit to be connected only to the local neighborhood of visible units, the SBM model can significantly reduce the number of trainable parameters while maintaining or improving the model performance.
2. **Image applications**: The paper pays special attention to image modeling and defines the connection method between hidden units and visible units by using the pixel spatial neighborhood relationship of images.
3. **Experimental verification**: Through experiments on multiple image datasets, the superior performance of the SBM model in image denoising and classification tasks has been verified.
### Experimental setup
- **Datasets**: The paper uses multiple benchmark datasets, including MNIST, FashionMNIST, OrganAMNIST, OrganCMNIST and OrganSMNIST, which contain 28×28 grayscale images.
- **Model structure**: The SBM models and the corresponding RBM models (called "twin RBMs") under different parameter settings are compared to ensure that both have the same number of hidden units for a fair comparison.
### Experimental results
- **Number of parameters**: The number of parameters of the SBM model is only 6% to 10% of that of the corresponding RBM model, which significantly reduces the computational cost.
- **Performance**: The SBM model shows a faster and more stable training process in image denoising and classification tasks, and achieves better results than RBM on some datasets.
In conclusion, by introducing the SBM model, this paper successfully solves the problem of too many parameters of the traditional RBM when processing high - dimensional data, while maintaining the high performance of the model, providing new solutions for fields such as image processing.