Contractive De-noising Auto-encoder

Fu-qiang Chen,Yan Wu,Guo-dong Zhao,Jun-ming Zhang,Ming Zhu,Jing Bai

DOI: https://doi.org/10.48550/arXiv.1305.4076

2014-04-23

Abstract:Auto-encoder is a special kind of neural network based on reconstruction. De-noising auto-encoder (DAE) is an improved auto-encoder which is robust to the input by corrupting the original data first and then reconstructing the original input by minimizing the reconstruction error function. And contractive auto-encoder (CAE) is another kind of improved auto-encoder to learn robust feature by introducing the Frobenius norm of the Jacobean matrix of the learned feature with respect to the original input. In this paper, we combine de-noising auto-encoder and contractive auto- encoder, and propose another improved auto-encoder, contractive de-noising auto- encoder (CDAE), which is robust to both the original input and the learned feature. We stack CDAE to extract more abstract features and apply SVM for classification. The experiment result on benchmark dataset MNIST shows that our proposed CDAE performed better than both DAE and CAE, proving the effective of our method.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to combine the advantages of the denoising auto - encoder (DAE) and the contractive auto - encoder (CAE) to propose a new auto - encoder model - the contractive denoising auto - encoder (CDAE), in order to improve the robustness of the model to input data and learned features and achieve better performance in classification tasks.

Specifically:

1. **Denoising auto - encoder (DAE)**: By introducing noise into the input data and then attempting to reconstruct the original input from the noisy data, the robustness of the model to the input data is improved.
2. **Contractive auto - encoder (CAE)**: By adding the Frobenius norm of the Jacobian matrix as a regularization term in the loss function, the learned features are made more stable and robust.
3. **Contractive denoising auto - encoder (CDAE)**: It combines the advantages of DAE and CAE. It can not only introduce noise into the input data to improve robustness, but also add a regularization term in the loss function to stabilize the learned features.

To verify the effectiveness of CDAE, the author conducted experiments using the MNIST handwritten digit dataset and used the extracted features for support vector machine (SVM) classification. The experimental results show that the performance of CDAE in classification tasks is better than that of traditional auto - encoders, DAE, and CAE.

In summary, this paper aims to propose a more powerful auto - encoder model CDAE by combining the advantages of DAE and CAE, in order to enhance the model's ability to handle noisy data and extract stable features, thereby achieving better results in classification tasks.