Abstract:The interpretability of models has become a crucial issue in Machine Learning because of algorithmic decisions' growing impact on real-world applications. Tree ensemble methods, such as Random Forests or XgBoost, are powerful learning tools for classification tasks. However, while combining multiple trees may provide higher prediction quality than a single one, it sacrifices the interpretability property resulting in "black-box" models. In light of this, we aim to develop an interpretable representation of a tree-ensemble model that can provide valuable insights into its behavior. First, given a target tree-ensemble model, we develop a hierarchical visualization tool based on a heatmap representation of the forest's feature use, considering the frequency of a feature and the level at which it is selected as an indicator of importance. Next, we propose a mixed-integer linear programming (MILP) formulation for constructing a single optimal multivariate tree that accurately mimics the target model predictions. The goal is to provide an interpretable surrogate model based on oblique hyperplane splits, which uses only the most relevant features according to the defined forest's importance indicators. The MILP model includes a penalty on feature selection based on their frequency in the forest to further induce sparsity of the splits. The natural formulation has been strengthened to improve the computational performance of {mixed-integer} software. Computational experience is carried out on benchmark datasets from the UCI repository using a state-of-the-art off-the-shelf solver. Results show that the proposed model is effective in yielding a shallow interpretable tree approximating the tree-ensemble decision function.

Frame Selection Based On Mixtures Of Trees In Discrete Data

New Fusional Framework Combining Sparse Selection and Clustering for Key Frame Extraction.

Sparse Variable Selection on High Dimensional Heterogeneous Data With Tree Structured Responses

Guaranteed Scalable Learning of Latent Tree Models

Learning High-Dimensional Markov Forest Distributions: Analysis of Error Rates

Dendrogram of mixing measures: Hierarchical clustering and model selection for finite mixture models

A Feature Matching Method: Sparse Feature Tree

Analyze Additive and Interaction Effects via Collaborative Trees

Data Driven Composite Shape Descriptor Design for Shape Retrieval with a VoR-Tree

Unboxing Tree Ensembles for interpretability: a hierarchical visualization tool and a multivariate optimal re-built tree

Modelling multivariate extreme value distributions via Markov trees

Segmentation of Individual Tree Points by Combining Marker-Controlled Watershed Segmentation and Spectral Clustering Optimization

Learning Causal Structure on Mixed Data with Tree-Structured Functional Models.

Optimal Sparse Regression Trees

Divide, Conquer, Combine Bayesian Decision Tree Sampling

Optimal Mixed Integer Linear Optimization Trained Multivariate Classification Trees

Finding Dependency Trees from Binary Data

An Efficient Framework for Exact Set Similarity Search Using Tree Structure Indexes.

Fast Sparse Decision Tree Optimization via Reference Ensembles

Ranking Perspective for Tree-based Methods with Applications to Symbolic Feature Selection

Efficient Tree Classifiers for Large Scale Datasets.