Abstract:Ensemble learning is a method that leverages weak learners to produce a strong learner. However, obtaining a large number of base learners requires substantial time and computational resources. Therefore, it is meaningful to study how to achieve the performance typically obtained with many base learners using only a few. We argue that to achieve this, it is essential to enhance both classification performance and generalization ability during the ensemble process. To increase model accuracy, each weak base learner needs to be more efficiently integrated. It is observed that different base learners exhibit varying levels of accuracy in predicting different classes. To capitalize on this, we introduce confidence tensors $\tilde{\mathbf{\Theta}}$ and $\tilde{\mathbf{\Theta}}_{rst}$ signifies the degree of confidence that the $t$-th base classifier assigns the sample to class $r$ while it actually belongs to class $s$. To the best of our knowledge, this is the first time an evaluation of the performance of base classifiers across different classes has been proposed. The proposed confidence tensor compensates for the strengths and weaknesses of each base classifier in different classes, enabling the method to achieve superior results with a smaller number of base learners. To enhance generalization performance, we design a smooth and convex objective function that leverages the concept of margin, making the strong learner more discriminative. Furthermore, it is proved that in gradient matrix of the loss function, the sum of each column's elements is zero, allowing us to solve a constrained optimization problem using gradient-based methods. We then compare our algorithm with random forests of ten times the size and other classical methods across numerous datasets, demonstrating the superiority of our approach.

Snapshot boosting: a fast ensemble framework for deep neural networks

Snapshot Ensembles: Train 1, get M for free

SSE: A New Selective Initialization Strategy for Snapshot Ensembling

Efficient Diversity-Driven Ensemble for Deep Neural Networks

Boost Neural Networks by Checkpoints

Deep InterBoost networks for small-sample image classification

Deep Boosting of Diverse Experts

Bagging and Boosting Fine-tuning for Ensemble Learning

Effective training-time stacking for ensembling of deep neural networks

Ensemble of Fast Learning Stochastic Gradient Boosting.

Auto-Ensemble: An Adaptive Learning Rate Scheduling based Deep Learning Model Ensembling

Improving Adversarial Robustness for Free with Snapshot Ensemble

A Neural Network Boosting Regression Model Based on XGBoost.

Snapshot Distillation: Teacher-Student Optimization in One Generation

Boosting and Other Ensemble Methods

BoostTree and BoostForest for Ensemble Learning

Snapshot Spectral Clustering -- a costless approach to deep clustering ensembles generation

Making Large Ensemble of Convolutional Neural Networks via Bootstrap Re-sampling

Improving Deep Neural Network Ensembles Using Reconstruction Error

Achieving More with Less: A Tensor-Optimization-Powered Ensemble Method

BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning