Abstract:Background. Clustering analysis discovers hidden structures in a data set by partitioning them into disjoint clusters. Robust accuracy measures that evaluate the goodness of clustering results are critical for algorithm development and model diagnosis. Common problems of current clustering accuracy measures include overlooking unmatched clusters, biases towards excessive clusters, unstable baselines, and difficult interpretation. In this study, we presented a novel accuracy measure, J-score, that addresses these issues. Methods. Given a data set with known class labels, J-score quantifies how well the hypothetical clusters produced by clustering analysis recover the true classes. It starts with bidirectional set matching to identify the correspondence between true classes and hypothetical clusters based on Jaccard index. It then computes two weighted sums of Jaccard indices measuring the reconciliation from classes to clusters and vice versa. The final J-score is the harmonic mean of the two weighted sums. Results. Via simulation studies, we evaluated the performance of J-score and compared with existing measures. Our results show that J-score is effective in distinguishing partition structures that differ only by unmatched clusters, rewarding correct inference of class numbers, addressing biases towards excessive clusters, and having a relatively stable baseline. The simplicity of its calculation makes the interpretation straightforward. It is a valuable tool complementary to other accuracy measures. We released an R/jScore package implementing the algorithm.

A score function for Bayesian cluster analysis

Information-Theoretic Scoring Rules to Learn Additive Bayesian Network Applied to Epidemiology

J-Score: A Robust Measure of Clustering Accuracy

Resolution Limit in Statistical Independence and Bayesian Network Scoring Functions

Optimal $F$-score Clustering for Bipartite Record Linkage

Nonparametric Analysis of Clustered Multivariate Data

Bayesian Rank-Clustering

Ensemble of classifiers based on score function defined by clusters and decision boundary of linear base learners

A Bayesian cluster validity index

Optimal Bayesian estimators for latent variable cluster models

Objective Bayesian inference with proper scoring rules

Bayesian model comparison with the Hyvärinen score: computation and consistency

Asymptotic Results for the Estimation of the Quadratic Score of a Clustering

Research on Bayesian Network Structure Score Function

Generalized Score Matching

Bayesian mixtures of common factor analyzers: Model, variational inference, and applications

Bayesian Cramér-Rao Bound Estimation with Score-Based Models

Search Algorithms and Loss Functions for Bayesian Clustering

A review on Bayesian model-based clustering

Optimal score estimation via empirical Bayes smoothing