Abstract:Ensemble learning is a method that leverages weak learners to produce a strong learner. However, obtaining a large number of base learners requires substantial time and computational resources. Therefore, it is meaningful to study how to achieve the performance typically obtained with many base learners using only a few. We argue that to achieve this, it is essential to enhance both classification performance and generalization ability during the ensemble process. To increase model accuracy, each weak base learner needs to be more efficiently integrated. It is observed that different base learners exhibit varying levels of accuracy in predicting different classes. To capitalize on this, we introduce confidence tensors $\tilde{\mathbf{\Theta}}$ and $\tilde{\mathbf{\Theta}}_{rst}$ signifies the degree of confidence that the $t$-th base classifier assigns the sample to class $r$ while it actually belongs to class $s$. To the best of our knowledge, this is the first time an evaluation of the performance of base classifiers across different classes has been proposed. The proposed confidence tensor compensates for the strengths and weaknesses of each base classifier in different classes, enabling the method to achieve superior results with a smaller number of base learners. To enhance generalization performance, we design a smooth and convex objective function that leverages the concept of margin, making the strong learner more discriminative. Furthermore, it is proved that in gradient matrix of the loss function, the sum of each column's elements is zero, allowing us to solve a constrained optimization problem using gradient-based methods. We then compare our algorithm with random forests of ten times the size and other classical methods across numerous datasets, demonstrating the superiority of our approach.

Generating Pool of Classifiers with Hyper-Parameter Optimization for Ensemble

Support Vector Machines Ensemble With Optimizing Weights By Genetic Algorithm

Optimizing Weights by Genetic Algorithm for Neural Network Ensemble

A Method of Adaptive Hyperparameter Optimization for Deep Generative Models

Optimizing Ensemble Weights and Hyperparameters of Machine Learning Models for Regression Problems

On Optimizing Ensemble Models using Column Generation

Integration of Classifier Diversity Measures for Feature Selection-Based Classifier Ensemble Reduction.

A New Rotation Forest Ensemble Algorithm

Generating Diverse and Accurate Classifier Ensembles Using Multi-Objective Optimization

Selective Ensemble Based on Transformation of Classifiers Used Spca

Rethinking Density Ratio Estimation Based Hyper-Parameter Optimization

Achieving More with Less: A Tensor-Optimization-Powered Ensemble Method

Reusing Genetic Programming for Ensemble Selection in Classification of Unbalanced Data

Ensemble Classifier Design Based on Perturbation Binary Salp Swarm Algorithm for Classification

Pooling homogeneous ensembles to build heterogeneous ones

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

Exploration of Classification Confidence in Ensemble Learning.

Greedy optimization classifiers ensemble based on diversity

Heuristic Parallel Selective Ensemble Algorithm Based on Clustering and Improved Simulated Annealing

Probabilistic SVM classifier ensemble selection based on GMDH-type neural network

Fast Hyperparameter Optimization of Deep Neural Networks via Ensembling Multiple Surrogates.