Abstract:How to obtain a model with good interpretability and performance has always been an important research topic. In this paper, we propose rectified decision trees (ReDT), a knowledge distillation based decision trees rectification with high interpretability, small model size, and empirical soundness. Specifically, we extend the impurity calculation and the pure ending condition of the classical decision tree to propose a decision tree extension that allows the use of soft labels generated by a well-trained teacher model in training and prediction process. It is worth noting that for the acquisition of soft labels, we propose a new multiple cross-validation based method to reduce the effects of randomness and overfitting. These approaches ensure that ReDT retains excellent interpretability and even achieves fewer nodes than the decision tree in the aspect of compression while having relatively good performance. Besides, in contrast to traditional knowledge distillation, back propagation of the student model is not necessarily required in ReDT, which is an attempt of a new knowledge distillation approach. Extensive experiments are conducted, which demonstrates the superiority of ReDT in interpretability, compression, and empirical soundness.

RST-Based Method for Constructing and Pruning Decision Trees

Rough set based decision tree

RST in Decision Tree Pruning

A new decision tree pruning method based on RST

New method for constructing decision tree based on rough sets theory

Rough set based approach to selection of node

A novel method for pruning decision trees

Novel Approach to Decision-Tree Construction.

The Method of Attribute Reduction Based on Decision Tree

Vprsm Based Decision Tree Classifier

Research about Decision Tree Algorithm Based on Attribute Reduction Technology

A New Pruning Method for Solving Decision Trees and Game Trees

An Optimized Algorithm of Decision Tree Based on Rough Sets Model

An improved decision tree algorithm based on boundary mixed attribute dependency

Attribute Reduction In Decision-Theoretic Rough Set Model: A Further Investigation

Minimum Query Set for Decision Tree Construction

Improved Algorithm of Decision Trees Construction Based on Variable Precision Rough Set

Rectified Decision Trees: Towards Interpretability, Compression and Empirical Soundness

A New Multivariate Decision Tree Construction Algorithm Based on Variable Precision Rough Set

Decision Tree Construction Algorithm Based on Multiscale Rough Set Model