Abstract:Among many solutions to the high-dimensional approximate nearest neighbor (ANN) search problem, locality sensitive hashing (LSH) is known for its sub-linear query time and robust theoretical guarantee on query accuracy. Traditional LSH methods can generate a small number of candidates quickly from hash tables but suffer from large index sizes and hash boundary problems. Recent studies to address these issues often incur extra overhead to identify eligible candidates or remove false positives, making query time no longer sub-linear. To address this dilemma, in this paper we propose a novel LSH scheme called DB-LSH which supports efficient ANN search for large high-dimensional datasets. It organizes the projected spaces with multi-dimensional indexes rather than using fixed-width hash buckets. Our approach can significantly reduce the space cost as by avoiding the need to maintain many hash tables for different bucket sizes. During the query phase of DB-LSH, a small number of high-quality candidates can be generated efficiently by dynamically constructing query-based hypercubic buckets with the required widths through index-based window queries. For a dataset of $n$ $d$-dimensional points with approximation ratio $c$, our rigorous theoretical analysis shows that DB-LSH achieves a smaller query cost ${O(n^{\rho^*} d\log n)}$, where ${\rho^*}$ is bounded by ${1/c^{\alpha}}$ while the bound is ${1/c}$ in the existing work. An extensive range of experiments on real-world data demonstrates the superiority of DB-LSH over state-of-the-art methods on both efficiency and accuracy.

LDC: Enabling Search By Partial Distance In A Hyper-Dimensional Space

Ldc: Enabling Search By Partial Distance In A Hyper-Dimensional Space

Accelerating Exact Nearest Neighbor Search in High Dimensional Euclidean Space Via Block Vectors

LBD:Exploring Local Bit-code Difference for KNN Search in High-dimensional Spaces

Composite Distance Transformation for Indexing and K -Nearest-neighbor Searching in High-Dimensional Spaces

Indexing High-Dimensional Data in Dual Distance Spaces

Fast k-Nearest Neighbour Search via Prioritized DCI

Indexing high-dimensional data in dual distance spaces: a symmetrical encoding approach

Dsh: Data Sensitive Hashing For High-Dimensional K-Nn Search

BC-iDistance:Bit-code Based Optimal High-dimensional Index

An Encoding-Based Dual Distance Tree High-Dimensional Index

Lazylsh: Approximate Nearest Neighbor Search For Multiple Distance Functions With A Single Index

Exploring Bit-Difference for Approximate KNN Search in High-Dimensional Databases

BC-iDistance： an optimized high-dimensional index for KNN processing

Fast answering k-nearest-neighbor queries over large image databases using dual distance transformation

Contorting High Dimensional Data for Efficient Main Memory KNN Processing

DForest: A Minimal Dimensionality-Aware Indexing for High-Dimensional Exact Similarity Search

Learning to Search Efficiently in High Dimensions.

Novel High-Dimensional Indexing Structure Based on Dual-Distance Metric

Preserving-Ignoring Transformation Based Index for Approximate k Nearest Neighbor Search

DB-LSH: Locality-Sensitive Hashing with Query-based Dynamic Bucketing