Abstract:Decision Tree is a well understood Machine Learning model that is based on minimizing impurities in the internal nodes. The most common impurity measures are Shannon entropy and Gini impurity. These impurity measures are insensitive to the order of training data and hence the final tree obtained is invariant to any permutation of the data. This is a limitation in terms of modeling when there are temporal order dependencies between data instances. In this research, we propose the adoption of Effort-To-Compress (ETC) - a complexity measure, for the first time, as an alternative impurity measure. Unlike Shannon entropy and Gini impurity, structural impurity based on ETC is able to capture order dependencies in the data, thus obtaining potentially different decision trees for different permutations of the same data instances, a concept we term as Permutation Decision Trees (PDT). We then introduce the notion of Permutation Bagging achieved using permutation decision trees without the need for random feature selection and sub-sampling. We conduct a performance comparison between Permutation Decision Trees and classical decision trees across various real-world datasets, including Appendicitis, Breast Cancer Wisconsin, Diabetes Pima Indian, Ionosphere, Iris, Sonar, and Wine. Our findings reveal that PDT demonstrates comparable performance to classical decision trees across most datasets. Remarkably, in certain instances, PDT even slightly surpasses the performance of classical decision trees. In comparing Permutation Bagging with Random Forest, we attain comparable performance to Random Forest models consisting of 50 to 1000 trees, using merely 21 trees. This highlights the efficiency and effectiveness of Permutation Bagging in achieving comparable performance outcomes with significantly fewer trees.

Learning Decision Trees Using Confusion Entropy

Dynamic Split-point Selection Method for Decision Tree Evolved by Gene Expression Programming

Unifying Decision Trees Split Criteria Using Tsallis Entropy

Improving Decision Trees by Tsallis Entropy Information Metric Method.

A less-greedy two-term Tsallis Entropy Information Metric approach for decision tree classification

Unifying Attribute Splitting Criteria of Decision Trees by Tsallis Entropy.

Evidential Decision Tree Based on Belief Entropy

Rank Entropy-Based Decision Trees for Monotonic Classification

A New Method for Learning Decision Tree Classifier

GB-CENT: Gradient Boosted Categorical Embedding and Numerical Trees.

A Classification Tree Method Based on Belief Entropy for Evidential Data

A New Approach of the Split Criteria of Decision Trees Based on the T-entropy

Learning decision trees through Monte Carlo tree search: An empirical evaluation

NeC4.5: Neural Ensemble Based C4.5

Interpretability Diversity for Decision-Tree-Initialized Dendritic Neuron Model Ensemble

Belief Entropy Tree and Random Forest: Learning from Data with Continuous Attributes and Evidential Labels

Permutation Decision Trees

Structural Diversity for Decision Tree Ensemble Learning

MTForest: Ensemble Decision Trees based on Multi-Task Learning

The Max-Cut Decision Tree: Improving on the Accuracy and Running Time of Decision Trees

Improved Information Gain Estimates for Decision Tree Induction