Abstract:Probabilistic error/loss performance evaluation instruments that are originally used for regression and time series forecasting are also applied in some binary-class or multi-class classifiers, such as artificial neural networks. This study aims to systematically assess probabilistic instruments for binary classification performance evaluation using a proposed two-stage benchmarking method called BenchMetrics Prob. The method employs five criteria and fourteen simulation cases based on hypothetical classifiers on synthetic datasets. The goal is to reveal specific weaknesses of performance instruments and to identify the most robust instrument in binary classification problems. The BenchMetrics Prob method was tested on 31 instrument/instrument variants, and the results have identified four instruments as the most robust in a binary classification context: Sum Squared Error ( SSE ), Mean Squared Error ( MSE ), Root Mean Squared Error ( RMSE , as the variant of MSE ), and Mean Absolute Error ( MAE ). As SSE has lower interpretability due to its [0, ∞) range, MAE in [0, 1] is the most convenient and robust probabilistic metric for generic purposes. In classification problems where large errors are more important than small errors, RMSE may be a better choice. Additionally, the results showed that instrument variants with summarization functions other than mean ( e.g. , median and geometric mean), LogLoss , and the error instruments with relative/percentage/symmetric-percentage subtypes for regression, such as Mean Absolute Percentage Error ( MAPE ), Symmetric MAPE ( sMAPE ), and Mean Relative Absolute Error ( MRAE ), were less robust and should be avoided. These findings suggest that researchers should employ robust probabilistic metrics when measuring and reporting performance in binary classification problems.

Mean squared error of prediction (MSEP) estimates for principal component regression (PCR) and partial least squares regression (PLSR)

Comparison of Regularized Least-Squares Approach and Predictive Error Method: Case Studies * *this Work is Supported by the National Science Foundation of China (no. 61273191) and by 973 Program of China (no. 2012CB720500). Corresponding Author Yucai Zhu. Tel. +15381085960. Fax +0571-87953353.

Calibration modeling by partial least-squares and principal component regression and its optimization using an improved leverage correction for prediction testing

Unbiased prediction errors for partial least squares regression models: Choosing a representative error estimator for process monitoring

Robust methods for partial least squares regression

Alternative Mean Square Error Estimators and Confidence Intervals for Prediction of Nonlinear Small Area Parameters

An adaptive strategy to improve the partial least squares model via minimum covariance determinant

A Novel Support Vector Regression Algorithm Incorporated with Prior Knowledge and Error Compensation for Small Datasets

BenchMetrics Prob: benchmarking of probabilistic error/loss performance evaluation instruments for binary classification problems

A better measure of relative prediction accuracy for model selection and model estimation

On a Biased Prediction Based on Optimal Mean Square Error Criterion

A note on the prediction error of principal component regression in high dimensions

A simulation study of the model evaluation criterion mmre

Prediction-Oriented Model Selection In Partial Least Squares Path Modeling

Genomic selection using principal component regression

Partial Least Squares Methods: Partial Least Squares Correlation and Partial Least Square Regression

A test of significance for partial least squares regression

Minimax Optimal Regression over Sobolev Spaces via Laplacian Eigenmaps on Neighborhood Graphs

Sketching for Principal Component Regression

The mean squared prediction error paradox

Prediction of multivariate responses with a select number of principal components