Abstract:Decision Tree is a well understood Machine Learning model that is based on minimizing impurities in the internal nodes. The most common impurity measures are Shannon entropy and Gini impurity. These impurity measures are insensitive to the order of training data and hence the final tree obtained is invariant to any permutation of the data. This is a limitation in terms of modeling when there are temporal order dependencies between data instances. In this research, we propose the adoption of Effort-To-Compress (ETC) - a complexity measure, for the first time, as an alternative impurity measure. Unlike Shannon entropy and Gini impurity, structural impurity based on ETC is able to capture order dependencies in the data, thus obtaining potentially different decision trees for different permutations of the same data instances, a concept we term as Permutation Decision Trees (PDT). We then introduce the notion of Permutation Bagging achieved using permutation decision trees without the need for random feature selection and sub-sampling. We conduct a performance comparison between Permutation Decision Trees and classical decision trees across various real-world datasets, including Appendicitis, Breast Cancer Wisconsin, Diabetes Pima Indian, Ionosphere, Iris, Sonar, and Wine. Our findings reveal that PDT demonstrates comparable performance to classical decision trees across most datasets. Remarkably, in certain instances, PDT even slightly surpasses the performance of classical decision trees. In comparing Permutation Bagging with Random Forest, we attain comparable performance to Random Forest models consisting of 50 to 1000 trees, using merely 21 trees. This highlights the efficiency and effectiveness of Permutation Bagging in achieving comparable performance outcomes with significantly fewer trees.

Multi-path Decision Tree

Dynamic Split-point Selection Method for Decision Tree Evolved by Gene Expression Programming

Rough set based decision tree

DTRadar: Accelerating Search Process of Decision Trees in Packet Classification.

MTForest: Ensemble Decision Trees based on Multi-Task Learning

Novel Design of Decision-Tree-Based Support Vector Machines Multi-Class Classifier

A linear multivariate decision tree with branch-and-bound components

Learning decision trees through Monte Carlo tree search: An empirical evaluation

A Communication-Efficient Parallel Algorithm for Decision Tree

A New Method for Learning Decision Tree Classifier

Decision Tree Support Vector Machine Based on Genetic Algorithm for Multi-Class Classification

Permutation Decision Trees

Building decision trees for the multi-class imbalance problem

The Max-Cut Decision Tree: Improving on the Accuracy and Running Time of Decision Trees

Heuristic Mode Research and Application of Decision Tree Algorithm

Decision tree methods: applications for classification and prediction

Process Decision Tree Model Based on Multi-Dimensional Time Series

Tree in Tree: from Decision Trees to Decision Graphs

Multi-grained and multi-layered gradient boosting decision tree for credit scoring

An Explainable Bayesian Decision Tree Algorithm

Interpretable Decision Tree Search as a Markov Decision Process