What problem does this paper attempt to address?

The problem that this paper attempts to solve is: on large - scale datasets, the existing dendrogram construction methods in single - linkage clustering are inefficient, especially when dealing with highly skewed dendrograms, and these methods are difficult to parallelize. Specifically, the traditional agglomerative or divisive techniques for dendrogram construction on the minimum spanning tree (MST) cannot be efficiently parallelized on multi - thread accelerators such as GPU, especially in the case of skewed dendrograms which are common in real - data. ### Specific Problem Description 1. **Limitations of Existing Methods**: - Traditional methods such as the agglomerative method and the divisive method perform poorly when dealing with skewed dendrograms. - These methods are difficult to be efficiently parallelized on GPU, resulting in a long computing time. - In large - scale low - dimensional datasets, dendrogram construction becomes the main bottleneck, taking up most of the computing time. 2. **Challenges in Practical Applications**: - For large - scale datasets in fields such as astronomy and bioinformatics, the time consumption of dendrogram construction is too high, which affects the performance of the overall algorithm. - Especially in the Hdbscan* algorithm, MST construction can be efficiently completed on GPU, but dendrogram construction still depends on CPU, resulting in a performance bottleneck. ### Goals of the Paper To solve the above problems, the paper proposes the Pandora algorithm, which aims to efficiently construct the dendrogram of single - linkage clustering through a novel recursive tree - contraction method, and this algorithm is fully parallelized and suitable for running on multi - thread accelerators such as GPU. Specific goals include: - **Improve Parallelization Efficiency**: Ensure that all steps can be efficiently executed in a multi - thread environment and fully utilize the powerful computing power of GPU. - **Handle Skewed Dendrograms**: The algorithm can effectively handle highly skewed dendrograms and avoid the performance degradation of traditional methods when dealing with such structures. - **Optimize Computational Complexity**: Through the recursive tree - contraction method, simplify the initial dendrogram construction process, and then gradually reconstruct the complete dendrogram to ensure the optimal computational complexity of the algorithm in the worst - case scenario. ### Main Contributions - Propose a novel parallel dendrogram construction algorithm based on tree - contraction. - Analyze the necessary and sufficient conditions of edge - contraction techniques. - Derive the asymptotic lower - bound complexity of any dendrogram construction algorithm and prove the work - optimality of the Pandora algorithm. - Provide an efficient performance - portable implementation that supports three different architectures (including GPU), significantly improving the computing speed of the Hdbscan* algorithm on GPU. Through these improvements, the Pandora algorithm can significantly improve the efficiency of dendrogram construction on large - scale datasets, especially achieving a 15 - 37 - fold speed - up on GPU.

PANDORA: A Parallel Dendrogram Construction Algorithm for Single Linkage Clustering on GPU

MSKD: multi-split KD-tree design on GPU

Optimal Parallel Algorithms for Dendrogram Computation and Single-Linkage Clustering

A New Linkless Hierarchical Structure Based on Perfect Hashing

Fast Parallel Algorithms for Euclidean Minimum Spanning Tree and Hierarchical Spatial Clustering

Parallel Filtered Graphs for Hierarchical Clustering

A Parallel Varied Density-Based Clustering Algorithm with Optimized Data Partition

A Complete Linkage Algorithm for Clustering Dynamic Datasets

Faster Parallel Exact Density Peaks Clustering

Paralinear Distance and Its Algorithm for Hierarchical Clustering of High-dimensional Discrete Variables

Genie: A new, fast, and outlier-resistant hierarchical clustering algorithm

PECANN: Parallel Efficient Clustering with Graph-Based Approximate Nearest Neighbor Search

cuSLINK: Single-linkage Agglomerative Clustering on the GPU

Mining Arbitrary Shaped Clusters and Outputting a High Quality Dendrogram.

A Parallel Algorithm for Exact Bayesian Structure Discovery in Bayesian Networks

Order preserving hierarchical agglomerative clustering

Parallel Massive Clustering of Discrete Distributions

Streaming Hierarchical Clustering Based on Point-Set Kernel

A Surprisingly Simple Method for Distributed Euclidean-Minimum Spanning Tree / Single Linkage Dendrogram Construction from High Dimensional Embeddings via Distance Decomposition

Faster Parallel Triangular Maximally Filtered Graphs and Hierarchical Clustering

Data Structures & Algorithms for Exact Inference in Hierarchical Clustering