Abstract:Multiple metrics are used when assessing and validating the performance of quantitative structure–activity relationship (QSAR) models. In the case of binary classification, balanced accuracy is a metric to assess the global performance of such models. In contrast to accuracy, balanced accuracy does not depend on the respective prevalence of the two categories in the test set that is used to validate a QSAR classifier. As such, balanced accuracy is used to overcome the effect of imbalanced test sets on the model's perceived accuracy. Matthews' correlation coefficient (MCC), an alternative global performance metric, is also known to mitigate the imbalance of the test set. However, in contrast to the balanced accuracy, MCC remains dependent on the respective prevalence of the predicted categories. For simplicity, the rest of this work is based on the positive prevalence. The MCC value may be underestimated at high or extremely low positive prevalence. It contributes to more challenging comparisons between experiments using test sets with different positive prevalences and may lead to incorrect interpretations. The concept of balanced metrics beyond balanced accuracy is, to the best of our knowledge, not yet described in the cheminformatic literature. Therefore, after describing the relevant literature, this manuscript will first formally define a confusion matrix, sensitivity and specificity and then present, with synthetic data, the danger of comparing performance metrics under nonconstant prevalence. Second, it will demonstrate that balanced accuracy is the performance metric accuracy calibrated to a test set with a positive prevalence of 50% (i.e., balanced test set). This concept of balanced accuracy will then be extended to the MCC after showing its dependency on the positive prevalence. Applying the same concept to any other performance metric and widening it to the concept of calibrated metrics will then be briefly discussed. We will show that, like balanced accuracy, any balanced performance metric may be expressed as a function of the well-known values of sensitivity and specificity. Finally, a tale of two MCCs will exemplify the use of this concept of balanced MCC versus MCC with four use cases using synthetic data.

A statistical comparison between Matthews correlation coefficient (MCC), prevalence threshold, and Fowlkes-Mallows index

The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation

The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation

The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification

Asymptotic Properties of Matthews Correlation Coefficient

Pearson-Matthews correlation coefficients for binary and multinary classification and hypothesis testing

The MCC approaches the geometric mean of precision and recall as true negatives approach infinity

Empirical Comparison of Area under ROC curve (AUC) and Mathew Correlation Coefficient (MCC) for Evaluating Machine Learning Algorithms on Imbalanced Datasets for Binary Classification

Mind your prevalence!

Pearson–Matthews correlation coefficients for binary and multinary classification

EBT: a Statistic Test Identifying Moderate Size of Significant Features with Balanced Power and Precision for Genome-Wide Rate Comparisons

Prevalence Threshold and bounds in the Accuracy of Binary Classification Systems

Why Cohen’s Kappa should be avoided as performance measure in classification

The MCC-F1 curve: a performance evaluation technique for binary classification

A unifying view for performance measures in multi-class prediction

High Agreement and High Prevalence: The Paradox of Cohen’s Kappa

Analysis and Comparison of Classification Metrics

Never mind the metrics -- what about the uncertainty? Visualising confusion matrix metric distributions

Comparing Accuracy Assessments to Infer Superiority of Image Classification Methods

BenchMetrics: a systematic benchmarking method for binary classification performance metrics