Abstract:This paper introduces a novel formulation of the clustering problem, namely, the minimum sum-of-squares clustering of infinitely tall data (MSSC-ITD), and presents HPClust, an innovative set of hybrid parallel approaches for its effective solution. By utilizing modern high-performance computing techniques, HPClust enhances key clustering metrics: effectiveness, computational efficiency, and scalability. In contrast to vanilla data parallelism, which only accelerates processing time through the MapReduce framework, our approach unlocks superior performance by leveraging the multi-strategy competitive–cooperative parallelism and intricate properties of the objective function landscape. Unlike other available algorithms that struggle to scale, our algorithm is inherently parallel in nature, improving solution quality through increased scalability and parallelism and outperforming even advanced algorithms designed for small- and medium-sized datasets. Our evaluation of HPClust, featuring four parallel strategies, demonstrates its superiority over traditional and cutting-edge methods by offering better performance in the key metrics. These results also show that parallel processing not only enhances the clustering efficiency, but the accuracy as well. Additionally, we explore the balance between computational efficiency and clustering quality, providing insights into optimal parallel strategies based on dataset specifics and resource availability. This research advances our understanding of parallelism in clustering algorithms, demonstrating that a judicious hybridization of advanced parallel approaches yields optimal results for MSSC-ITD. Experiments on the synthetic data further confirm HPClust's exceptional scalability and robustness to noise.

A High-Performance External Validity Index for Clustering with a Large Number of Clusters

An Extenics-Based Criteria Clustering Method

CNMBI: Determining the Number of Clusters Using Center Pairwise Matching and Boundary Filtering.

Accuracy Evaluation of Overlapping and Multi-resolution Clustering Algorithms on Large Datasets

Normalised clustering accuracy: An asymmetric external cluster validity measure

High-Performance Hybrid Algorithm for Minimum Sum-of-Squares Clustering of Infinitely Tall Data

OMES: a new evaluation strategy using optimal matching for document clustering.

An Effective and Efficient Approach for Clusterability Evaluation

Efficient synthetical clustering validity indexes for hierarchical clustering

SMMP: A Stable-Membership-Based Auto-Tuning Multi-Peak Clustering Algorithm.

Improved Hierarchical Clustering on Massive Datasets with Broad Guarantees

Scalable Unpaired Multi-View Clustering with Bipartite Graph Matching

Self-adaptive Method of Determining Optimal Number of Clusters in Kernel-based Clustering Algorithm

Performance evaluation of some clustering algorithms and validity indices

Discrete and Balanced Spectral Clustering with Scalability

A New Validity Index for Fuzzy-Possibilistic C-Means Clustering

Unique Metric for Health Analysis with Optimization of Clustering Activity and Cross Comparison of Results from Different Approach

Fully Scalable MPC Algorithms for Clustering in High Dimension

An Information-Theoretic External Cluster-Validity Measure

Near-Optimal Comparison Based Clustering