Abstract:Cost-sensitive feature selection describes a feature selection problem, where features raise individual costs for inclusion in a model. These costs allow to incorporate disfavored aspects of features, e.g. failure rates of as measuring device, or patient harm, in the model selection process. Random Forests define a particularly challenging problem for feature selection, as features are generally entangled in an ensemble of multiple trees, which makes a post hoc removal of features infeasible. Feature selection methods therefore often either focus on simple pre-filtering methods, or require many Random Forest evaluations along their optimization path, which drastically increases the computational complexity. To solve both issues, we propose Shallow Tree Selection, a novel fast and multivariate feature selection method that selects features from small tree structures. Additionally, we also adapt three standard feature selection algorithms for cost-sensitive learning by introducing a hyperparameter-controlled benefit-cost ratio criterion (BCR) for each method. In an extensive simulation study, we assess this criterion, and compare the proposed methods to multiple performance-based baseline alternatives on four artificial data settings and seven real-world data settings. We show that all methods using a hyperparameterized BCR criterion outperform the baseline alternatives. In a direct comparison between the proposed methods, each method indicates strengths in certain settings, but no one-fits-all solution exists. On a global average, we could identify preferable choices among our BCR based methods. Nevertheless, we conclude that a practical analysis should never rely on a single method only, but always compare different approaches to obtain the best results.

Nonparametric feature selection by random forests and deep neural networks

Improved Parallel Random Forest Algorithm Combining Information Theory and Norm

Hierarchical Feature Selection for Random Projection

A New Noisy Random Forest Based Method for Feature Selection

A Novel Random Forests-Based Feature Selection Method for Microarray Expression Data Analysis

An Improvement to Feature Selection of Random Forests on Spark

The All Relevant Feature Selection using Random Forest

High-dimensional EEG Features Selection Based on Random Forests Algorithm

A Novel Feature Subspace Selection Method in Random Forests for High Dimensional Data.

Cost-sensitive feature selection using random forest: Selecting low-cost subsets of informative features

Stratified sampling for feature subspace selection in random forests for high dimensional data

Feature Selection of Non-Intrusive Load Monitoring System Using RFE and RF

Feature Selection Methods for Cost-Constrained Classification in Random Forests

A Model-Free Feature Selection Technique of Feature Screening and Random Forest-Based Recursive Feature Elimination

A Deep Neural Network Model using Random Forest to Extract Feature Representation for Gene Expression Data Classification

A Feature Selection Method Based on Sparse Regularization Deep Neural Network

Distribution-dependent Feature Selection for Deep Neural Networks

KNCFS: Feature selection for high-dimensional datasets based on improved random multi-subspace learning

Unsupervised Feature Selection by Preserving Stochastic Neighbors

Feature Selection and Optimization of Random Forest Modeling

Towards the effectiveness of Deep Convolutional Neural Network based Fast Random Forest Classifier