Abstract:In numerous classification problems, class distribution is not balanced. For example, positive examples are rare in the fields of disease diagnosis and credit card fraud detection. General machine learning methods are known to be suboptimal for such imbalanced classification. One popular solution is to balance training data by oversampling the underrepresented (or undersampling the overrepresented) classes before applying machine learning algorithms. However, despite its popularity, the effectiveness of sampling has not been rigorously and comprehensively evaluated. This study assessed combinations of seven sampling methods and eight machine learning classifiers (56 varieties in total) using 31 datasets with varying degrees of imbalance. We used the areas under the precision-recall curve (AUPRC) and receiver operating characteristics curve (AUROC) as the performance measures. The AUPRC is known to be more informative for imbalanced classification than the AUROC. We observed that sampling significantly changed the performance of the classifier (paired t-tests P < 0.05) only for few cases (12.2% in AUPRC and 10.0% in AUROC). Surprisingly, sampling was more likely to reduce rather than improve the classification performance. Moreover, the adverse effects of sampling were more pronounced in AUPRC than in AUROC. Among the sampling methods, undersampling performed worse than others. Also, sampling was more effective for improving linear classifiers. Most importantly, we did not need sampling to obtain the optimal classifier for most of the 31 datasets. In addition, we found two interesting examples in which sampling significantly reduced AUPRC while significantly improving AUROC (paired t-tests P < 0.05). In conclusion, the applicability of sampling is limited because it could be ineffective or even harmful. Furthermore, the choice of the performance measure is crucial for decision making. Our results provide valuable insights into the effect and characteristics of sampling for imbalanced classification.

Solving the Data Imbalance Problem of P300 Detection Via Random Under-Sampling Bagging SVMs.

Imbalanced Data Sets Classification Method Based on Over-Sampling Technique

The effect of methods addressing the class imbalance problem on P300 detection

Epileptic Signal Classification Based on Synthetic Minority Oversampling and Blending Algorithm.

A Novel Imbalanced Data Classification Method Based on Weakly Supervised Learning for Fault Diagnosis

P300 feature extraction algorithm based on wavelet transform and temporal energy entropy

Fusion Algorithm for Imbalanced EEG Data Processing in Seizure Detection

Random Undersampling on Imbalance Time Series Data for Anomaly Detection

A novel algorithm to enhance P300 in single trials: application to lie detection using F-score and SVM

An empirical evaluation of sampling methods for the classification of imbalanced data

Imbalanced Data Classification Based on Improved Random-SMOTE and Feature Standard Deviation

Grouped-sampling technique to deal with unbalance in Raman spectral data modeling

Transfer Learning Algorithm of P300-EEG Signal Based on XDAWN Spatial Filter and Riemannian Geometry Classifier

RSBagging: An ensemble classifier detecting the after-effects of ischemic stroke through EEG connectivity and microstates

Research of P300 Feature Extraction Algorithm Based on ICA and Wavelet Transform

Sample and feature selecting based ensemble learning for imbalanced problems

A New Sampling Approach for Classification of Imbalanced Data Sets with High Density.

Alleviating Class Imbalance Problem in Automatic Sleep Stage Classification

A P300-Detection Method Based on Logistic Regression and a Convolutional Neural Network

An Impartial Semi-Supervised Learning Strategy for Imbalanced Classification on VHR Images

An Empirical Study on the Joint Impact of Feature Selection and Data Re-sampling on Imbalance Classification