Abstract:In numerous classification problems, class distribution is not balanced. For example, positive examples are rare in the fields of disease diagnosis and credit card fraud detection. General machine learning methods are known to be suboptimal for such imbalanced classification. One popular solution is to balance training data by oversampling the underrepresented (or undersampling the overrepresented) classes before applying machine learning algorithms. However, despite its popularity, the effectiveness of sampling has not been rigorously and comprehensively evaluated. This study assessed combinations of seven sampling methods and eight machine learning classifiers (56 varieties in total) using 31 datasets with varying degrees of imbalance. We used the areas under the precision-recall curve (AUPRC) and receiver operating characteristics curve (AUROC) as the performance measures. The AUPRC is known to be more informative for imbalanced classification than the AUROC. We observed that sampling significantly changed the performance of the classifier (paired t-tests P < 0.05) only for few cases (12.2% in AUPRC and 10.0% in AUROC). Surprisingly, sampling was more likely to reduce rather than improve the classification performance. Moreover, the adverse effects of sampling were more pronounced in AUPRC than in AUROC. Among the sampling methods, undersampling performed worse than others. Also, sampling was more effective for improving linear classifiers. Most importantly, we did not need sampling to obtain the optimal classifier for most of the 31 datasets. In addition, we found two interesting examples in which sampling significantly reduced AUPRC while significantly improving AUROC (paired t-tests P < 0.05). In conclusion, the applicability of sampling is limited because it could be ineffective or even harmful. Furthermore, the choice of the performance measure is crucial for decision making. Our results provide valuable insights into the effect and characteristics of sampling for imbalanced classification.

Uniform Sampling Table Method and Its Applications II—Evaluating the Uniform Sampling by Experiment

Stratified Uniform Sampling Method for Power System Reliability Evaluation

Uniform design and its applications in chemistry and chemical engineering

[Formulation and evaluation of a new content uniformity testing scheme by variables]

Determining Sample Size in Binary Measurement System

The Evaluation of Some Two-stage Sampling Methods

Application of Near Infrared Spectroscopy Combined with Competitive Adaptive Reweighted Sampling Partial Least Squares for On-Line Monitoring of the Concentration Process of Wangbi Tablets

Application of Uniform Design and Intelligence Optimization Software in Synthesis

Comparison of several variable selection methods for quantitative analysis and monitoring of the Yangxinshi tablet process using near-infrared spectroscopy

A novel mixture sampling strategy combining latin hypercube sampling with optimized one factor at a time method: A case study on mixtures of antibiotics and pesticides

Establishment and Validation of the Quantitative Analysis of Multi-Components by Single Marker for the Quality Control of Qishen Yiqi Dripping Pills by High-Performance Liquid Chromatography with Charged Aerosol Detection.

A Novel Sample Selection Strategy by Near-Infrared Spectroscopy-Based High Throughput Tablet Tester for Content Uniformity in Early-Phase Pharmaceutical Product Development.

Comparison of Testability Verification Test Sample Size Determination Methods Based on Binomial Distribution

Application and validation of a statistically derived risk-based sampling plan to improve efficiency of inspection and enforcement

Chemical Distribution Uniformity Assessment of "Intra-Tablet" by Hyperspectral Raman Imaging Analysis

A Practical Analysis Procedure on Generalizing Comparative Effectiveness in the Randomized Clinical Trial to the Real-world Trialeligible Population

Sample Size for Tablet Compression and Capsule Filling Events During Process Validation

An empirical evaluation of sampling methods for the classification of imbalanced data

A Method of Sampling Inspection

Sample Mass Estimate for the Use of Near-Infrared and Raman Spectroscopy to Monitor Content Uniformity in a Tablet Press Feed Frame of a Drug Product Continuous Manufacturing Process.

Random Sparse Sampling Strategy Using Stochastic Simulation and Estimation for a Population Pharmacokinetic Study