Abstract:Ensemble methods have been widely used for improving the results of the best single classificationmodel. A large body of works have achieved better performance mainly by applying one specific ensemble method. However, very few works have explored complex fusion schemes using het-erogeneous ensembles with new aggregation strategies. This paper is three-fold: 1) It provides an overview of the most popular ensemble methods, 2) analyzes several fusion schemes using MNIST as guiding thread and 3) introduces MNIST-NET10, a complex heterogeneous fusion architecture based on a degree of certainty aggregation approach; it combines two heterogeneous schemes from the perspective of data, model and fusion strategy. MNIST-NET10 reaches a new record in MNISTwith only 10 misclassified images. Our analysis shows that such complex heterogeneous fusionarchitectures based on the degree of certainty can be considered as a way of taking benefit fromdiversity.

What problem does this paper attempt to address?

This paper attempts to address the problem of further reducing the error rate in handwritten digit recognition tasks. Specifically, the paper proposes a complex heterogeneous fusion architecture, MNIST-NET10, which achieves an error rate of 0.1% by combining various deep learning models and data augmentation techniques, as well as a new aggregation strategy—the degree of certainty aggregation method. This means that only 10 out of 10,000 test images are misclassified. ### Main Contributions of the Paper: 1. **Reviewed Popular Ensemble Methods**: The paper first provides an overview of the most commonly used ensemble methods in machine learning. 2. **Provided the Latest Results on the MNIST Dataset**: The paper details several of the best-performing ensemble methods on the MNIST dataset. 3. **Introduced MNIST-NET10**: This is a complex heterogeneous fusion architecture that combines various models and data processing techniques, achieving a new record with only 10 misclassified images. ### Specific Methods of the Paper: 1. **Data Preprocessing**: - Multiple data augmentation techniques such as rotation, translation, elastic deformation, and Gaussian smoothing are used to generate multiple datasets. - CapsNet is used to preprocess the data, generating new representations. 2. **Fusion Scheme**: - **Weighted Fusion Based on High-Performance Classifiers**: Several networks that perform well on MNIST (such as DropConnect, MCDNN, CapsNet, etc.) are selected and fused through different combinations (such as parallel execution). - **Multi-Level Fusion Scheme**: Combines various heterogeneous ensemble methods such as ECOC (Error-Correcting Output Codes), Bagging, and Label Switching, and applies these methods at different levels. ### Experimental Results: - Through the above methods, MNIST-NET10 achieved an error rate of 0.1% on the MNIST dataset, significantly outperforming existing methods. ### Conclusion: The paper demonstrates that through a complex heterogeneous fusion architecture and a degree of certainty aggregation method, model diversity and performance can be effectively improved, leading to better results in handwritten digit recognition tasks.

MNIST-NET10: A heterogeneous deep networks fusion based on the degree of certainty to reach 0.1 error rate. Ensembles overview and proposal

Deep Neural Network Fusion via Graph Matching with Applications to Model Ensemble and Federated Learning.

Deep fusion network based ensemble learning approach for detection intensive classification

Deep Network Ensembles for Aerial Scene Classification

Accurate and Reliable Predictions with Mutual-Transport Ensemble

Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation

A fragmented neural network ensemble method and its application to image classification

BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning

Feature Fusion for Improved Classification: Combining Dempster-Shafer Theory and Multiple CNN Architectures

Enhancing Adaptive Deep Networks for Image Classification via Uncertainty-aware Decision Fusion

Fast and reliable uncertainty quantification with neural network ensembles for industrial image classification

Deep Ensemble Network Based on Multi-Path Fusion

Multi Loss-based Feature Fusion and Top Two Voting Ensemble Decision Strategy for Facial Expression Recognition in the Wild

A Study of Ensemble of Hybrid Networks with Strong Regularization

Hybrid Ensemble Construction with Selected Neural Networks.

Exploring Model Learning Heterogeneity for Boosting Ensemble Robustness

MM-Net: A MixFormer-Based Multi-Scale Network for Anatomical and Functional Image Fusion

On the Connection of Deep Fusion to Ensembling.

HatchEnsemble: an efficient and practical uncertainty quantification method for deep neural networks

United We Stand: Using Epoch-wise Agreement of Ensembles to Combat Overfit

Heterogeneous Multi-column ConvNets with a Fusion Framework for Object Recognition