Abstract:Approximate nearest neighbor search (ANNS) is a fundamental problem that has attracted widespread attention for decades. Multi-probe ANNS is one of the most important classes of ANNS methods, playing crucial roles in disk-based, GPU-based, and distributed scenarios. The state-of-the-art multi-probe ANNS approaches typically perform in a fixed-configuration manner. For example, each query is dispatched to a fixed number of partitions to run ANNS algorithms locally, and the results will be merged to obtain the final result set. Our observation shows that such fixed configurations typically lead to a non-optimal accuracy–efficiency trade-off. To further optimize multi-probe ANNS, we propose to generate efficient configurations for each query individually. By formalizing the per-query optimization as a 0–1 knapsack problem and its variants, we identify that the k NN distribution (the proportion of k nearest neighbors of a query placed in each partition) is essential to the optimization. Then we develop LEQAT (LEarned Query-Aware OpTimizer), which leverages k NN distribution to seek optimal configurations for each query. LEQAT comes with (i) a machine learning model to learn and estimate k NN distributions based on historical or sample queries and (ii) efficient query optimization algorithms to determine the partitions to probe and the number of searching neighbors in each partition. We apply LEQAT to three state-of-the-art ANNS methods IVF, HNSW, and SSG under clustering-based partitioning, evaluating the overall performance on several real-world datasets. The results show that LEQAT consistently reduces the latency by up to 58% and improves the throughput by up to 3.9 times.

Accelerating Massive Queries of Approximate Nearest Neighbor Search on High-Dimensional Data

Efficient Approximate Nearest Neighbor Search in Multi-dimensional Databases.

DiskANN++: Efficient Page-based Search over Isomorphic Mapped Graph Index using Query-sensitivity Entry Vertex

Learning-based Query Optimization for Multi-Probe Approximate Nearest Neighbor Search.

Towards Efficient Index Construction and Approximate Nearest Neighbor Search in High-Dimensional Spaces.

JUNO: Optimizing High-Dimensional Approximate Nearest Neighbour Search with Sparsity-Aware Algorithm and Ray-Tracing Core Mapping

Learned Probing Cardinality Estimation for High-Dimensional Approximate NN Search.

SRS: solving c-approximate nearest neighbor queries in high dimensional euclidean space with a tiny index

GPU-accelerated Proximity Graph Approximate Nearest Neighbor Search and Construction

An Efficient and Robust Framework for Approximate Nearest Neighbor Search with Attribute Constraint

FusionANNS: An Efficient CPU/GPU Cooperative Processing Architecture for Billion-scale Approximate Nearest Neighbor Search

Efficient Reverse $k$ Approximate Nearest Neighbor Search over High-Dimensional Vectors

SymphonyQG: Towards Symphonious Integration of Quantization and Graph for Approximate Nearest Neighbor Search

I/O Efficient Approximate Nearest Neighbour Search Based on Learned Functions.

Subspace Collision: An Efficient and Accurate Framework for High-dimensional Approximate Nearest Neighbor Search

Efficient Large-scale Approximate Nearest Neighbor Search on the GPU

Efficient Large-Scale Approximate Nearest Neighbor Search on OpenCL FPGA

Revisiting the Index Construction of Proximity Graph-Based Approximate Nearest Neighbor Search

GPU-utilized Composite Quantization for Large-scale High-dimension ANN Search

Efficient Nearest Neighbor Search in High Dimensional Hamming Space

Approximate Nearest Neighbor Search on High Dimensional Data — Experiments, Analyses, and Improvement