Abstract:Decades after their inception, regression forests continue to provide state-of-the-art accuracy, outperforming in this respect alternative machine learning models such as regression trees or even neural networks. However, being an ensemble method, the one aspect where regression forests tend to severely underperform regression trees is interpretability. In the present work, we revisit forest pruning, an approach that aims to have the best of both worlds: the accuracy of regression forests and the interpretability of regression trees. This pursuit, whose foundation lies at the core of random forest theory, has seen vast success in empirical studies. In this paper, we contribute theoretical results that support and qualify those empirical findings; namely, we prove the asymptotic advantage of a Lasso-pruned forest over its unpruned counterpart under extremely weak assumptions, as well as high-probability finite-sample generalization bounds for regression forests pruned according to the main methods, which we then validate by way of simulation. Then, we test the accuracy of pruned regression forests against their unpruned counterparts on 19 different datasets (16 synthetic, 3 real). We find that in the vast majority of scenarios tested, there is at least one forest-pruning method that yields equal or better accuracy than the original full forest (in expectation), while just using a small fraction of the trees. We show that, in some cases, the reduction in the size of the forest is so dramatic that the resulting sub-forest can be meaningfully merged into a single tree, obtaining a level of interpretability that is qualitatively superior to that of the original regression forest, which remains a black box.

A novel method for pruning decision trees

Class-Aware Pruning for Efficient Neural Networks

RST in Decision Tree Pruning

A new decision tree pruning method based on RST

A Feature-map Discriminant Perspective for Pruning Deep Neural Networks

Pruning algorithm of classificaiton and regression decision tree based on Akaike information criteria

A New Pruning Method for Solving Decision Trees and Game Trees

Cost-sensitive Decision Tree with Probabilistic Pruning Mechanism

RST-Based Method for Constructing and Pruning Decision Trees

FreePrune: an Automatic Pruning Framework Across Various Granularities Based on Training-Free Evaluation

Novel Pruning of Dendritic Neuron Models for Improved System Implementation and Performance

Pruning Decision Tree Using Genetic Algorithms

Research and Improvement of Decision Tree's Prune Algorithm

New method for constructing decision tree based on rough sets theory

A Comprehensive Study of Structural Pruning for Vision Models

Random Forest Pruning Techniques: A Recent Review

Pruning Foundation Models for High Accuracy without Retraining

Theoretical and Empirical Advances in Forest Pruning

An Analysis of Reduced Error Pruning

A New Classification Method to Overcome Over-Branching