Abstract:Random Forests have been one of the most popular bagging methods in the past few decades, especially due to their success at handling tabular datasets. They have been extensively studied and compared to boosting models, like XGBoost, which are generally considered more performant. Random Forests adopt several simplistic assumptions, such that all samples and all trees that form the forest are equally important for building the final model. We introduce Enhanced Random Forests, an extension of vanilla Random Forests with extra functionalities and adaptive sample and model weighting. We develop an iterative algorithm for adapting the training sample weights, by favoring the hardest examples, and an approach for finding personalized tree weighting schemes for each new sample. Our method significantly improves upon regular Random Forests across 15 different binary classification datasets and considerably outperforms other tree methods, including XGBoost, when run with default hyperparameters, which indicates the robustness of our approach across datasets, without the need for extensive hyperparameter tuning. Our tree-weighting methodology results in enhanced or comparable performance to the uniformly weighted ensemble, and is, more importantly, leveraged to define importance scores for trees based on their contributions to classifying each new sample. This enables us to only focus on a small number of trees as the main models that define the outcome of a new sample and, thus, to partially recover interpretability, which is critically missing from both bagging and boosting methods. In binary classification problems, the proposed extensions and the corresponding results suggest the equivalence of bagging and boosting methods in performance, and the edge of bagging in interpretability by leveraging a few learners of the ensemble, which is not an option in the less explainable boosting methods.

A binary decision tree implementation of a boosted strong classifier

Z-Adaboost: Boosting 2-thresholded weak classifiers for object detection

Boosting 2-Thresholded Weak Classifiers over Scattered Rectangle Features for Object Detection.

To Boost or Not to Boost? On the Limits of Boosted Trees for Object Detection

Multi-class AdaBoost ELM

Fast and Robust Object Detection Using Asymmetric Totally Corrective Boosting

Improved AdaBoost.M1 of Decision Trees with Confidence-Rated Predictions

Binary Classification: Is Boosting stronger than Bagging?

A Theory of Probabilistic Boosting, Decision Trees and Matryoshki

GAdaBoost: Accelerating Adaboost Feature Selection with Genetic Algorithms

Compact Multi-Class Boosted Trees

LACBoost and FisherBoost: Optimally Building Cascade Classifiers

On the Design of Cascades of Boosted Ensembles for Face Detection

BVDT: A Boosted Vector Decision Tree Algorithm for Multi-Class Classification Problems

The return of AdaBoost.MH: multi-class Hamming trees

Asymmetric Totally-Corrective Boosting for Real-Time Object Detection

A Study of AdaBoost with Naive Bayesian Classifiers: Weakness and Improvement.

FastBDT: A speed-optimized and cache-friendly implementation of stochastic gradient-boosted decision trees for multivariate classification

Tree-Structured Boosting: Connections Between Gradient Boosted Stumps and Full Decision Trees

A Revised AdaBoost Algorithm——AD AdaBoost

Extremely Fast Decision Tree