Abstract:Cost-sensitive feature selection describes a feature selection problem, where features raise individual costs for inclusion in a model. These costs allow to incorporate disfavored aspects of features, e.g. failure rates of as measuring device, or patient harm, in the model selection process. Random Forests define a particularly challenging problem for feature selection, as features are generally entangled in an ensemble of multiple trees, which makes a post hoc removal of features infeasible. Feature selection methods therefore often either focus on simple pre-filtering methods, or require many Random Forest evaluations along their optimization path, which drastically increases the computational complexity. To solve both issues, we propose Shallow Tree Selection, a novel fast and multivariate feature selection method that selects features from small tree structures. Additionally, we also adapt three standard feature selection algorithms for cost-sensitive learning by introducing a hyperparameter-controlled benefit-cost ratio criterion (BCR) for each method. In an extensive simulation study, we assess this criterion, and compare the proposed methods to multiple performance-based baseline alternatives on four artificial data settings and seven real-world data settings. We show that all methods using a hyperparameterized BCR criterion outperform the baseline alternatives. In a direct comparison between the proposed methods, each method indicates strengths in certain settings, but no one-fits-all solution exists. On a global average, we could identify preferable choices among our BCR based methods. Nevertheless, we conclude that a practical analysis should never rely on a single method only, but always compare different approaches to obtain the best results.

A cost sensitive decision tree algorithm with two adaptive mechanisms

Rough set based decision tree

Handling Over-Fitting in Test Cost-Sensitive Decision Tree Learning by Feature Selection, Smoothing and Pruning

Cost-Sensitive Classification Using Decision Trees, Boosting and MetaCost

Decision-Theoretic Rough Set and Cost-Sensitive Classification

Heuristic Mode Research and Application of Decision Tree Algorithm

An adaptive Cost-sensitive Classifier

SVM-Based Cost-sensitive Classification Algorithm with Error Cost and Class-dependent Reject Cost

The Max-Cut Decision Tree: Improving on the Accuracy and Running Time of Decision Trees

Multi-scale decision systems with test cost and applications to three-way multi-attribute decision-making

Cost-Sensitive Classification: Empirical Evaluation of a Hybrid Genetic Decision Tree Induction Algorithm

Taiga: Performance Optimization of the C4.5 Decision Tree Construction Algorithm

Anytime Induction of Low-cost, Low-error Classifiers: a Sampling-based Approach

Cost-sensitive feature selection using random forest: Selecting low-cost subsets of informative features

Boosting Cost-Sensitive Trees

Early Warning of Enterprise Financial Risk Based on Decision Tree Algorithm

A Decision Support System Using Two-Level Classifier for Smart Grid

A Multiphase Cost-Sensitive Learning Method Based on the Multiclass Three-Way Decision-Theoretic Rough Set Model

Heuristic Mode Research Of Decision Tree Algorithm Based On Synthesizing Effect

Feature Selection Methods for Cost-Constrained Classification in Random Forests