Abstract:Despite their remarkable effectiveness and broad application, the drivers of success underlying ensembles of trees are still not fully understood. In this paper, we highlight how interpreting tree ensembles as adaptive and self-regularizing smoothers can provide new intuition and deeper insight to this topic. We use this perspective to show that, when studied as smoothers, randomized tree ensembles not only make predictions that are quantifiably more smooth than the predictions of the individual trees they consist of, but also further regulate their smoothness at test-time based on the dissimilarity between testing and training inputs. First, we use this insight to revisit, refine and reconcile two recent explanations of forest success by providing a new way of quantifying the conjectured behaviors of tree ensembles objectively by measuring the effective degree of smoothing they imply. Then, we move beyond existing explanations for the mechanisms by which tree ensembles improve upon individual trees and challenge the popular wisdom that the superior performance of forests should be understood as a consequence of variance reduction alone. We argue that the current high-level dichotomy into bias- and variance-reduction prevalent in statistics is insufficient to understand tree ensembles -- because the prevailing definition of bias does not capture differences in the expressivity of the hypothesis classes formed by trees and forests. Instead, we show that forests can improve upon trees by three distinct mechanisms that are usually implicitly entangled. In particular, we demonstrate that the smoothing effect of ensembling can reduce variance in predictions due to noise in outcome generation, reduce variability in the quality of the learned function given fixed input data and reduce potential bias in learnable functions by enriching the available hypothesis space.

Variable Randomness in Decision Tree Ensembles

Maximizing tree diversity by building complete-random decision trees

Spectrum of Variable-Random Trees

Structural Diversity for Decision Tree Ensemble Learning

A New Rotation Forest Ensemble Algorithm

On Computing Optimal Tree Ensembles

Selective Ensemble Of Decision Trees

Random boosting and random^2 forests -- A random tree depth injection approach

Ensemble of optimal trees, random forest and random projection ensemble classification

MTForest: Ensemble Decision Trees based on Multi-Task Learning

Ensembles of Probabilistic Regression Trees

Inherently Interpretable Tree Ensemble Learning

BoostTree and BoostForest for Ensemble Learning

Exogenous Randomness Empowering Random Forests

Boosting-Based Sequential Meta-Tree Ensemble Construction for Improved Decision Trees

Extreme Learning Tree

The random subspace method for constructing decision forests

Differentially- and non-differentially-private random decision trees

Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers

Born-Again Tree Ensembles

A New Random Forest Ensemble of Intuitionistic Fuzzy Decision Trees