Abstract:Deep learning has attracted a lot of attention in research and industry in recent years. Behind the success of deep learning, there is much space for improvement. It is difficult to identify if a testing sample can be represented by the deep network effectively before we examining the final result. In this paper, we proposed a dynamic boosting strategy according to reconstruction error in deep networks. We use reconstruction error to determine whether the result is reliable or not. From the perspective of prediction interval, we demonstrated that with the increase of reconstruction error, the prediction interval would become bigger. Therefore, the classification result is not reliable when the reconstruction error exceeds the predetermined threshold. Since we can record the reconstruction error as well as the classification error for all training samples in training set. We can learn an extra boosting model besides the deep network in training set to improve the performance of the model. An important factor in learning the boosting model is to determine an appropriate threshold for selecting training samples. In testing, we first examine whether the reconstruction error of a testing sample exceeds the threshold to determine if we should use the boosting model. If the boosting model is used, the final result is the average of the output of the deep network and the boosting model. We conducted experiments on two widely used classification datasets and an air quality dataset. From the experiments, we see that our boosting strategy is effective in improving the performance of classification. We tested several boosting models in this paper. They can all reduce the test error to some extent under appropriate parameter settings.

Improving Deep Neural Network Ensembles Using Reconstruction Error

Optimizing Weights by Genetic Algorithm for Neural Network Ensemble

Dynamically Weighted Ensemble Neural Networks with Generalized Regression Neural Network for Solving Regression Problems

Dynamically Weighted Ensemble Neural Networks for Regression Problems

Deep Ensembles Work, But Are They Necessary?

Ensembling Neural Networks: Many Could Be Better Than All

“In-Network Ensemble”: Deep Ensemble Learning with Diversified Knowledge Distillation

A DNN-Ensemble Method for Error Reduction and Training Data Selection in DNN Based Modeling

Boost Neural Networks by Checkpoints

Deep Neural Network Ensembles against Deception: Ensemble Diversity, Accuracy and Robustness

Auto-Ensemble: An Adaptive Learning Rate Scheduling based Deep Learning Model Ensembling

Remaining useful life estimation based on selective ensemble of deep neural networks with diversity

Dynamic Boosting in Deep Learning Using Reconstruction Error.

Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity

Neural Subnetwork Ensembles

Deep Ensembles: A Loss Landscape Perspective

The Diversified Ensemble Neural Network.

Ensemble deep learning: A review

Making Large Ensemble of Convolutional Neural Networks via Bootstrap Re-sampling

Ex uno plures: Splitting One Model into an Ensemble of Subnetworks

Efficient Diversity-Driven Ensemble for Deep Neural Networks