Abstract:This paper introduces Weighted Optimal Classification Forests (WOCFs), a new family of classifiers that takes advantage of an optimal ensemble of decision trees to derive accurate and interpretable classifiers. We propose a novel mathematical optimization-based methodology which simultaneously constructs a given number of trees, each of them providing a predicted class for the observations in the feature space. The classification rule is derived by assigning to each observation its most frequently predicted class among the trees. We provide a mixed integer linear programming formulation (MIP) for the problem and several novel MIP strengthening / scaling techniques. We report the results of our computational experiments, from which we conclude that our method has equal or superior performance compared with state-of-the-art tree-based classification methods for small to medium-sized instances. We also present three real-world case studies showing that our methodology has very interesting implications in terms of interpretability. Overall, WOCFs complement existing methods such as CART, Optimal Classification Trees, Random Forests and XGBoost. In addition to its Pareto improvement on accuracy and interpretability, we also see unique properties emerging in terms of different trees focusing on different feature variables. This provides nontrivial improvement in interpretability and usability of the trained model in terms of counterfactual explanation. Thus, despite the apparent computational challenge of WOCFs that limit the size of the problems that can be efficiently solved with current MIP, this is an important research direction that can lead to qualitatively different insights for researchers and complement the toolbox of practitioners for high stakes problems.

A Projection Pursuit Forest Algorithm for Supervised Classification

A New Rotation Forest Ensemble Algorithm

Classifiability-Based Optimal Discriminatory Projection Pursuit

A Forest of Trees with Principal Direction Specified Oblique Split on Random Subspace.

Interactive Graphics for Visually Diagnosing Forest Classifiers in R

Ensemble of optimal trees, random forest and random projection ensemble classification

A Novel Feature Subspace Selection Method in Random Forests for High Dimensional Data.

Proximity Forest: An effective and scalable distance-based classifier for time series

Random Similarity Forests

Random Geometric Prior Forest for Multiclass Object Segmentation

Random Subsequence Forests

A Mathematical Programming Approach to Optimal Classification Forests

Random Projections for Classification: A Recovery Approach

Riemann-Lebesgue Forest for Regression

Head Pose Estimation Based on Random Forests for Multiclass Classification

Random Planted Forest: a directly interpretable tree ensemble

Random-projection ensemble classification

Conformalized Semi-supervised Random Forest for Classification and Abnormality Detection

ggRandomForests: Visually Exploring a Random Forest for Regression

MTForest: Ensemble Decision Trees based on Multi-Task Learning

Accelerated and Interpretable Oblique Random Survival Forests