Abstract:Decades after their inception, regression forests continue to provide state-of-the-art accuracy, outperforming in this respect alternative machine learning models such as regression trees or even neural networks. However, being an ensemble method, the one aspect where regression forests tend to severely underperform regression trees is interpretability. In the present work, we revisit forest pruning, an approach that aims to have the best of both worlds: the accuracy of regression forests and the interpretability of regression trees. This pursuit, whose foundation lies at the core of random forest theory, has seen vast success in empirical studies. In this paper, we contribute theoretical results that support and qualify those empirical findings; namely, we prove the asymptotic advantage of a Lasso-pruned forest over its unpruned counterpart under extremely weak assumptions, as well as high-probability finite-sample generalization bounds for regression forests pruned according to the main methods, which we then validate by way of simulation. Then, we test the accuracy of pruned regression forests against their unpruned counterparts on 19 different datasets (16 synthetic, 3 real). We find that in the vast majority of scenarios tested, there is at least one forest-pruning method that yields equal or better accuracy than the original full forest (in expectation), while just using a small fraction of the trees. We show that, in some cases, the reduction in the size of the forest is so dramatic that the resulting sub-forest can be meaningfully merged into a single tree, obtaining a level of interpretability that is qualitatively superior to that of the original regression forest, which remains a black box.

Leaf Node-Level Ensemble Pruning Approaches Based On Node-Sample Correlation For Random Forest

Optimally Pruning Decision Tree Ensembles With Feature Cost

Margin optimization based pruning for random forest

Optimizing the number of branches in a decision forest using association rule metrics

ForestPrune: Compact Depth-Controlled Tree Ensembles

Improving the Accuracy-Memory Trade-Off of Random Forests Via Leaf-Refinement

Theoretical and Empirical Advances in Forest Pruning

Trees, Forests, Chickens, and Eggs: When and Why to Prune Trees in a Random Forest

Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensembles

A Probabilistic Ensemble Pruning Algorithm

A Novel Bayesian Ensemble Pruning Method

Margin-Based Pareto Ensemble Pruning: an Ensemble Pruning Algorithm That Learns to Search Optimized Ensembles.

Random Forest Pruning Techniques: A Recent Review

A New Rotation Forest Ensemble Algorithm

Random Decision Dag: An Entropy Based Compression Approach For Random Forest

Predictive Ensemble Pruning by Expectation Propagation

Ensemble Pruning Based on Frequent Patterns

On Extreme Pruning of Random Forest Ensembles for Real-time Predictive Applications

RST in Decision Tree Pruning

Margin distribution and structural diversity guided ensemble pruning

Improving deep forest by ensemble pruning based on feature vectorization and quantum walks