Abstract:Methods for voting classification algorithms, such as Bagging and AdaBoost, have been shown to be very successful in improving the accuracy of certain classifiers for artificial and real-world datasets. We review these algorithms and describe a large empirical study comparing several variants in conjunction with a decision tree inducer (three variants) and a Naive-Bayes inducer. The purpose of the study is to improve our understanding of why and when these algorithms, which use perturbation, reweighting, and combination techniques, affect classification error. We provide a bias and variance decomposition of the error to show how different methods and variants influence these two terms. This allowed us to determine that Bagging reduced variance of unstable methods, while boosting methods (AdaBoost and Arc-x4) reduced both the bias and variance of unstable methods but increased the variance for Naive-Bayes, which was very stable. We observed that Arc-x4 behaves differently than AdaBoost if reweighting is used instead of resampling, indicating a fundamental difference. Voting variants, some of which are introduced in this paper, include: pruning versus no pruning, use of probabilistic estimates, weight perturbations (Wagging), and backfitting of data. We found that Bagging improves when probabilistic estimates in conjunction with no-pruning are used, as well as when the data was backfit. We measure tree sizes and show an interesting positive correlation between the increase in the average tree size in AdaBoost trials and its success in reducing the error. We compare the mean-squared error of voting methods to non-voting methods and show that the voting methods lead to large and significant reductions in the mean-squared errors. Practical problems that arise in implementing boosting algorithms are explored, including numerical instabilities and underflows. We use scatterplots that graphically show how AdaBoost reweights instances, emphasizing not only “hard” areas but also outliers and noise.

Improved AdaBoost Algorithm Based on Vote Entropy

An Enhanced Adaboost Algorithm With Naive Bayesian Text Categorization Based On A Novel Re-Weighting Strategy

An Improved Boosting Algorithm with Naive Bayesian Classification Based on Multi-Views

Lda Boost Classification: Boosting by Topics

An Improved FloatBoost Algorithm for Naïve Bayes Text Classification.

An improved floatboost algorithm for Naïve bayes text classification

A Study of AdaBoost with Naive Bayesian Classifiers: Weakness and Improvement.

The improved AdaBoost algorithms for imbalanced data classification

An Improved AdaBoost Algorithm for Unbalanced Classification Data.

Improved boosting algorithm through weighted k-nearest neighbors classifier

Multiclass Boosting With Adaptive Group-Based Knn And Its Application In Text Categorization

Improved Centroid Classifier Based on AdaBoost. MR for Text Categorization

Edited AdaBoost by Weighted Knn

Improved Stumps Combined by Boosting for Text Categorization

An Improved AdaBoost Algorithm Based on Adaptive Weight Adjusting

Effective Boosting of Naïve Bayesian Classifiers by Local Accuracy Estimation

Improved KNN Algorithm based on Probability and Adaptive K Value.

Improved Behavior-Based Malware Detection Algorithm with AdaBoost

An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants

Improved Adaboost Algorithm for Classification Based on Noise Confidence Degree and Weighted Feature Selection

Improved AdaBoost.M1 of Decision Trees with Confidence-Rated Predictions