Abstract:Clustering is a fundamental task in machine learning. One of the most successful and broadly used algorithms is DBSCAN, a density-based clustering algorithm. DBSCAN requires $\epsilon$-nearest neighbor graphs of the input dataset, which are computed with range-search algorithms and spatial data structures like KD-trees. Despite many efforts to design scalable implementations for DBSCAN, existing work is limited to low-dimensional datasets, as constructing $\epsilon$-nearest neighbor graphs is expensive in high-dimensions. In this paper, we modify DBSCAN to enable use of $\kappa$-nearest neighbor graphs of the input dataset. The $\kappa$-nearest neighbor graphs are constructed using approximate algorithms based on randomized projections. Although these algorithms can become inaccurate or expensive in high-dimensions, they possess a much lower memory overhead than constructing $\epsilon$-nearest neighbor graphs. We delineate the conditions under which $k$NN-DBSCAN produces the same clustering as DBSCAN. We also present an efficient parallel implementation of the overall algorithm using OpenMP for shared memory and MPI for distributed memory parallelism. We present results on up to 16 billion points in 20 dimensions, and perform weak and strong scaling studies using synthetic data. Our code is efficient in both low and high dimensions. We can cluster one billion points in 3D in less than one second on 28K cores on the Frontera system at the Texas Advanced Computing Center (TACC). In our largest run, we cluster 65 billion points in 20 dimensions in less than 40 seconds using 114,688 x86 cores on TACC's Frontera system. Also, we compare with a state of the art parallel DBSCAN code; on 20d/4M point dataset, our code is up to 37$\times$ faster.

Contorting High Dimensional Data for Efficient Main Memory KNN Processing

Adaptive Quantization Of The High-Dimensional Data For Efficient Knn Processing

Composite Distance Transformation for Indexing and K -Nearest-neighbor Searching in High-Dimensional Spaces

Indexing high-dimensional data for efficient in-memory similarity search

Diagonal Ordering: A New Approach to High-Dimensional KNN Processing.

Indexing High-Dimensional Data in Dual Distance Spaces

Indexing high-dimensional data in dual distance spaces: a symmetrical encoding approach

Efficient index-based KNN join processing for high-dimensional data

iDistance: An adaptive B+-tree based indexing method for nearest neighbor search

An Encoding-Based Dual Distance Tree High-Dimensional Index

Optimizing Kd-Trees for Scalable Visual Descriptor Indexing

A Clustered Dwarf Structure to Speed Up Queries on Data Cubes

Exploring Bit-Difference for Approximate KNN Search in High-Dimensional Databases

Multidimensional binary search trees used for associative searching

Hierarchical indexing scheme for fast search in a large-scale image database

KNN-DBSCAN: a DBSCAN in high dimensions

Making the Pyramid Technique Robust to Query Types and Workloads

Fast answering k-nearest-neighbor queries over large image databases using dual distance transformation

Efficient Nearest Neighbor Search Using Dynamic Programming

Dimensionality Reduced Clustered Data and Order Partition and Stepwise Dimensionality Increasing Indices

Trinary-projection trees for approximate nearest neighbor search.