Abstract:Decision trees are among the most popular machine learning models and are used routinely in applications ranging from revenue management and medicine to bioinformatics. In this paper, we consider the problem of learning optimal binary classification trees with univariate splits. Literature on the topic has burgeoned in recent years, motivated both by the empirical suboptimality of heuristic approaches and the tremendous improvements in mixed-integer optimization (MIO) technology. Yet, existing MIO-based approaches from the literature do not leverage the power of MIO to its full extent: they rely on weak formulations, resulting in slow convergence and large optimality gaps. To fill this gap in the literature, we propose an intuitive flow-based MIO formulation for learning optimal binary classification trees. Our formulation can accommodate side constraints to enable the design of interpretable and fair decision trees. Moreover, we show that our formulation has a stronger linear optimization relaxation than existing methods in the case of binary data. We exploit the decomposable structure of our formulation and max-flow/min-cut duality to derive a Benders' decomposition method to speed-up computation. We propose a tailored procedure for solving each decomposed subproblem that provably generates facets of the feasible set of the MIO as constraints to add to the main problem. We conduct extensive computational experiments on standard benchmark datasets on which we show that our proposed approaches are 29 times faster than state-of-the-art MIO-based techniques and improve out-of-sample performance by up to 8%.

Algorithm for Constructing Fast Classification Trees

FROD: an Efficient Framework for Optimizing Decision Trees in Packet Classification

DTRadar: Accelerating Search Process of Decision Trees in Packet Classification.

DOT: Towards Fast Decision Tree Packet Classification by Optimizing Rule Partitions

Taiga: Performance Optimization of the C4.5 Decision Tree Construction Algorithm

Efficient Tree Classifiers for Large Scale Datasets.

Nonlinear Optimal Classification Trees

Strong Optimal Classification Trees

Novel Design of Decision-Tree-Based Support Vector Machines Multi-Class Classifier

An Optimized Multi-class Classification Algorithm Based on SVM Decision Tree

Boosting Trees for Cost-Sensitive Classifications.

Towards Optimal Discriminating Order for Multiclass Classification

Study on GA-based SVM multi-class classification decision-tree optimization algorithm

A new fast reduction technique based on binary nearest neighbor tree.

An Optimized Algorithm of Decision Tree Based on Rough Sets Model

Research on Application of Decision Tree in Classifying Data

Mixed-Integer Linear Optimization for Semi-Supervised Optimal Classification Trees

An Improvement On The Algorithm Of Decision Tree

A Non-Collision Hash Trie-Tree Based Fast IP Classification Algorithm.

Classification and Recognition of Building Appearance Based on Optimized Gradient-Boosted Decision Tree Algorithm

An Improved Network Traffic Classification Algorithm Based on Hadoop Decision Tree