Abstract:Cost-sensitive feature selection describes a feature selection problem, where features raise individual costs for inclusion in a model. These costs allow to incorporate disfavored aspects of features, e.g. failure rates of as measuring device, or patient harm, in the model selection process. Random Forests define a particularly challenging problem for feature selection, as features are generally entangled in an ensemble of multiple trees, which makes a post hoc removal of features infeasible. Feature selection methods therefore often either focus on simple pre-filtering methods, or require many Random Forest evaluations along their optimization path, which drastically increases the computational complexity. To solve both issues, we propose Shallow Tree Selection, a novel fast and multivariate feature selection method that selects features from small tree structures. Additionally, we also adapt three standard feature selection algorithms for cost-sensitive learning by introducing a hyperparameter-controlled benefit-cost ratio criterion (BCR) for each method. In an extensive simulation study, we assess this criterion, and compare the proposed methods to multiple performance-based baseline alternatives on four artificial data settings and seven real-world data settings. We show that all methods using a hyperparameterized BCR criterion outperform the baseline alternatives. In a direct comparison between the proposed methods, each method indicates strengths in certain settings, but no one-fits-all solution exists. On a global average, we could identify preferable choices among our BCR based methods. Nevertheless, we conclude that a practical analysis should never rely on a single method only, but always compare different approaches to obtain the best results.

Distance Correlation-Based Feature Selection in Random Forest

Feature selection with distance correlation

Novel Feature Selection Algorithms Based on Crowding Distance and Pearson Correlation Coefficient

A Hybrid Feature Selection Approach by Correlation-Based Filters and SVM-RFE

A Comparative Study on the Use of Correlation Coefficients for Redundant Feature Elimination

Improved distance correlation estimation

Improving Effectiveness of Intrusion Detection by Correlation Feature Selection

A Feature Selection Method Using Conditional Correlation Dispersion and Redundancy Analysis

Modeling of Feature Selection Based on Random Forest Algorithm and Pearson Correlation Coefficient

RANDOM FOREST AND SUPPORT VECTOR MACHINE ON FEATURES SELECTION FOR REGRESSION ANALYSIS

Distance Rank Score: Unsupervised filter method for feature selection on imbalanced dataset

The extension of Pearson correlation coefficient, measuring noise, and selecting features

Speaker-independent Speech Emotion Recognition Based on Random Forest Feature Selection Algorithm

Feature redundancy removal for text classification using correlated feature subsets

R2CI: Information theoretic-guided feature selection with multiple correlations

Feature selection based on mutual information with correlation coefficient

Optimizing Performance Random Forest Algorithm Using Correlation-Based Feature Selection (CFS) Method to Improve Distributed Denial of Service (DDoS) Attack Detection Accuracy

Linear Correlation-Based Feature Selection for Network Intrusion Detection Model

Feature Selection Methods for Cost-Constrained Classification in Random Forests

A new filter feature selection algorithm for classification task by ensembling pearson correlation coefficient and mutual information