Abstract:Machine learning algorithms often contain many hyperparameters whose values affect the predictive performance of the induced models in intricate ways. Due to the high number of possibilities for these hyperparameter configurations and their complex interactions, it is common to use optimization techniques to find settings that lead to high predictive performance. However, insights into efficiently exploring this vast space of configurations and dealing with the trade-off between predictive and runtime performance remain challenging. Furthermore, there are cases where the default hyperparameters fit the suitable configuration. Additionally, for many reasons, including model validation and attendance to new legislation, there is an increasing interest in interpretable models, such as those created by the decision tree (DT) induction algorithms. This paper provides a comprehensive approach for investigating the effects of hyperparameter tuning for the two DT induction algorithms most often used, CART and C4.5. DT induction algorithms present high predictive performance and interpretable classification models, though many hyperparameters need to be adjusted. Experiments were carried out with different tuning strategies to induce models and to evaluate hyperparameters' relevance using 94 classification datasets from OpenML. The experimental results point out that different hyperparameter profiles for the tuning of each algorithm provide statistically significant improvements in most of the datasets for CART, but only in one-third for C4.5. Although different algorithms may present different tuning scenarios, the tuning techniques generally required few evaluations to find accurate solutions. Furthermore, the best technique for all the algorithms was the Irace. Finally, we found out that tuning a specific small subset of hyperparameters is a good alternative for achieving optimal predictive performance.

Matching Model Versus Single Model: A Study Of The Requirement To Match Class Distribution Using Decision Trees

Model Matching: A Novel Framework to Use Clustering Strategy to Solve the Classification Problem.

Learning When Training Data are Costly: The Effect of Class Distribution on Tree Induction

Boosting Trees for Cost-Sensitive Classifications.

Novel Design of Decision-Tree-Based Support Vector Machines Multi-Class Classifier

Model stability: a key factor in determining whether an algorithm produces an optimal model from a matching distribution

Model Combination In The Multiple-Data-Batches Scenario

Decision Combination Based on the Characterisation of Predictive Accuracy.

Matching provides efficient decisions

Building decision trees for the multi-class imbalance problem

Cost-Sensitive Classification Using Decision Trees, Boosting and MetaCost

A linear multivariate decision tree with branch-and-bound components

Matched Machine Learning: A Generalized Framework for Treatment Effect Inference With Learned Metrics

Better trees: an empirical study on hyperparameter tuning of classification decision tree induction algorithms

Matching Bounds: How Choice of Matching Algorithm Impacts Treatment Effects Estimates and What to Do about It

A New Method for Learning Decision Tree Classifier

The random subspace method for constructing decision forests

Decision tree modeling using R

Learning from Batched Data: Model Combination Versus Data Combination

Tree models for assessing covariate-dependent method agreement

Reconciling Model Multiplicity for Downstream Decision Making