Abstract:Under-bagging (UB), which combines under-sampling and bagging, is a popular ensemble learning method for training classifiers on an imbalanced data. Using bagging to reduce the increased variance caused by the reduction in sample size due to under-sampling is a natural approach. However, it has recently been pointed out that in generalized linear models, naive bagging, which does not consider the class imbalance structure, and ridge regularization can produce the same results. Therefore, it is not obvious whether it is better to use UB, which requires an increased computational cost proportional to the number of under-sampled data sets, when training linear models. Given such a situation, in this study, we heuristically derive a sharp asymptotics of UB and use it to compare with several other popular methods for learning from imbalanced data, in the scenario where a linear classifier is trained from a two-component mixture data. The methods compared include the under-sampling (US) method, which trains a model using a single realization of the under-sampled data, and the simple weighting (SW) method, which trains a model with a weighted loss on the entire data. It is shown that the performance of UB is improved by increasing the size of the majority class while keeping the size of the minority fixed, even though the class imbalance can be large, especially when the size of the minority class is small. This is in contrast to US, whose performance is almost independent of the majority class size. In this sense, bagging and simple regularization differ as methods to reduce the variance increased by under-sampling. On the other hand, the performance of SW with the optimal weighting coefficients is almost equal to UB, indicating that the combination of reweighting and regularization may be similar to UB.

Out-of-Bag Estimation of the Optimal Hyperparameter in SubBag Ensemble Method.

Investigating the Effect of Randomly Selected Feature Subsets on Bagging and Boosting

A Weighted Subspace Approach for Improving Bagging Performance

Cross-validatory Framework for Optimal Parameter Estimation of KPCA and KPLS Models

A Method of Adaptive Hyperparameter Optimization for Deep Generative Models

Research of the Applications of Out-of-bag Sample

Bagging by Design (on the Suboptimality of Bagging)

Hyperparameter Selection for Subsampling Bootstraps

Under-bagging Nearest Neighbors for Imbalanced Classification

An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization

A replica analysis of under-bagging

Using Boosting to prune Double-Bagging ensembles

On the Subbagging Estimation for Massive Data

Subspace Ensembles for Classification

Cascaded Algorithm-Selection and Hyper-Parameter Optimization with Extreme-Region Upper Confidence Bound Bandit

A Novel Genetic Algorithm for Subspace Based Subclasssifier Selection

Cascaded Algorithm Selection With Extreme-Region UCB Bandit

Under-sampling class imbalanced datasets by combining clustering analysis and instance selection

A Novel Bagging Approach for Variable Ranking and Selection Via a Mixed Importance Measure

Non-I.I.D. Multi-Instance Dimensionality Reduction by Learning a Maximum Bag Margin Subspace

Selective Feature Bagging of one-class classifiers for novelty detection in high-dimensional data