Abstract:Machine learning algorithms often contain many hyperparameters whose values affect the predictive performance of the induced models in intricate ways. Due to the high number of possibilities for these hyperparameter configurations and their complex interactions, it is common to use optimization techniques to find settings that lead to high predictive performance. However, insights into efficiently exploring this vast space of configurations and dealing with the trade-off between predictive and runtime performance remain challenging. Furthermore, there are cases where the default hyperparameters fit the suitable configuration. Additionally, for many reasons, including model validation and attendance to new legislation, there is an increasing interest in interpretable models, such as those created by the decision tree (DT) induction algorithms. This paper provides a comprehensive approach for investigating the effects of hyperparameter tuning for the two DT induction algorithms most often used, CART and C4.5. DT induction algorithms present high predictive performance and interpretable classification models, though many hyperparameters need to be adjusted. Experiments were carried out with different tuning strategies to induce models and to evaluate hyperparameters' relevance using 94 classification datasets from OpenML. The experimental results point out that different hyperparameter profiles for the tuning of each algorithm provide statistically significant improvements in most of the datasets for CART, but only in one-third for C4.5. Although different algorithms may present different tuning scenarios, the tuning techniques generally required few evaluations to find accurate solutions. Furthermore, the best technique for all the algorithms was the Irace. Finally, we found out that tuning a specific small subset of hyperparameters is a good alternative for achieving optimal predictive performance.

Optimal decision trees for the algorithm selection problem: integer programming based approaches

Gois: Optimal Item Selection With Generalized Cross-Selling Considerations

Necessary and Sufficient Conditions for Optimal Decision Trees using Dynamic Programming

Learning optimal decision trees using constraint programming

Better trees: an empirical study on hyperparameter tuning of classification decision tree induction algorithms

Mixed integer linear optimization formulations for learning optimal binary classification trees

Optimal Sparse Decision Trees

Optimal or Greedy Decision Trees? Revisiting their Objectives, Tuning, and Performance

Mixed-Integer Linear Optimization for Semi-Supervised Optimal Classification Trees

Optimal Mixed Integer Linear Optimization Trained Multivariate Classification Trees

Strong Optimal Classification Trees

bsnsing: A decision tree induction method based on recursive optimal boolean rule composition

Optimization over decision trees: a case study for the design of stable direct-current electricity networks

Quant-BnB: A Scalable Branch-and-Bound Method for Optimal Decision Trees with Continuous Features

Learning Optimal and Fair Decision Trees for Non-Discriminative Decision-Making

Multi-objective integer programming approaches for solving optimal feature selection problem: a new perspective on multi-objective optimization problems in SBSE.

BooleanOCT: Optimal Classification Trees based on multivariate Boolean Rules

A Portfolio Approach to Algorithm Selection for Discrete Time-Cost Trade-off Problem

A Decision-Making Tool for Algorithm Selection Based on a Fuzzy TOPSIS Approach to Solve Replenishment, Production and Distribution Planning Problems

Optimal Deterministic Algorithm Generation