Abstract:Ensemble learning is a method that leverages weak learners to produce a strong learner. However, obtaining a large number of base learners requires substantial time and computational resources. Therefore, it is meaningful to study how to achieve the performance typically obtained with many base learners using only a few. We argue that to achieve this, it is essential to enhance both classification performance and generalization ability during the ensemble process. To increase model accuracy, each weak base learner needs to be more efficiently integrated. It is observed that different base learners exhibit varying levels of accuracy in predicting different classes. To capitalize on this, we introduce confidence tensors $\tilde{\mathbf{\Theta}}$ and $\tilde{\mathbf{\Theta}}_{rst}$ signifies the degree of confidence that the $t$-th base classifier assigns the sample to class $r$ while it actually belongs to class $s$. To the best of our knowledge, this is the first time an evaluation of the performance of base classifiers across different classes has been proposed. The proposed confidence tensor compensates for the strengths and weaknesses of each base classifier in different classes, enabling the method to achieve superior results with a smaller number of base learners. To enhance generalization performance, we design a smooth and convex objective function that leverages the concept of margin, making the strong learner more discriminative. Furthermore, it is proved that in gradient matrix of the loss function, the sum of each column's elements is zero, allowing us to solve a constrained optimization problem using gradient-based methods. We then compare our algorithm with random forests of ten times the size and other classical methods across numerous datasets, demonstrating the superiority of our approach.

An empirical bias–variance analysis of DECORATE ensemble method at different training sample sizes

Ensembling over Classifiers: a Bias-Variance Perspective

On the Size of Training Set and the Benefit from Ensemble

A New Rotation Forest Ensemble Algorithm

Experimental Study and Comparison of Imbalance Ensemble Classifiers with Dynamic Selection Strategy

A Bias-Variance Decomposition for Ensembles over Multiple Synthetic Datasets

On the Insufficiency of the Large Margins Theory in Explaining the Performance of Ensemble Methods

Decorrelating Structure via Adapters Makes Ensemble Learning Practical for Semi-supervised Learning

Interpretability Diversity for Decision-Tree-Initialized Dendritic Neuron Model Ensemble

Learning to Diversify via Weighted Kernels for Classifier Ensemble

A Margin-Maximizing Fine-Grained Ensemble Method

Scalable Ensemble Diversification for OOD Generalization and Detection

Margin distribution and structural diversity guided ensemble pruning

Developing parsimonious ensembles using ensemble diversity within a reinforcement learning framework

DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation

Improving imbalance classification via ensemble learning based on two-stage learning

Leveraging Linear Independence of Component Classifiers: Optimizing Size and Prediction Accuracy for Online Ensembles

Achieving More with Less: A Tensor-Optimization-Powered Ensemble Method

A hybrid data-level ensemble to enable learning from highly imbalanced dataset

The Disparate Benefits of Deep Ensembles

An Exploration of How Training Set Composition Bias in Machine Learning Affects Identifying Rare Objects