Abstract:Given a vector dataset 𝒳, a query vector x⃗_q, graph-based Approximate Nearest Neighbor Search (ANNS) aims to build a proximity graph (PG) as an index of 𝒳 and approximately return vectors with minimum distances to x⃗_q by searching over the PG index. It suffers from the large-scale 𝒳 because a PG with full vectors is too large to fit into the memory, e.g., a billion-scale 𝒳 in 128 dimensions would consume nearly 600 GB memory. To solve this, Product Quantization (PQ) integrated graph-based ANNS is proposed to reduce the memory usage, using smaller compact codes of quantized vectors in memory instead of the large original vectors. Existing PQ methods do not consider the important routing features of PG, resulting in low-quality quantized vectors that affect the ANNS's effectiveness. In this paper, we present an end-to-end Routing-guided learned Product Quantization (RPQ) for graph-based ANNS. It consists of (1) a differentiable quantizer used to make the standard discrete PQ differentiable to suit for back-propagation of end-to-end learning, (2) a sampling-based feature extractor used to extract neighborhood and routing features of a PG, and (3) a multi-feature joint training module with two types of feature-aware losses to continuously optimize the differentiable quantizer. As a result, the inherent features of a PG would be embedded into the learned PQ, generating high-quality quantized vectors. Moreover, we integrate our RPQ with the state-of-the-art DiskANN and existing popular PGs to improve their performance. Comprehensive experiments on real-world large-scale datasets (from 1M to 1B) demonstrate RPQ's superiority, e.g., 1.7×-4.2× improvement on QPS at the same recall@10 of 95%.

Hierarchical quantization for billion-scale similarity retrieval on GPUs

Efficient Large-scale Approximate Nearest Neighbor Search on the GPU

Vector and Line Quantization for Billion-scale Similarity Search on GPUs

Large-scale Image Retrieval Based on Boosting Iterative Quantization Hashing with Query-Adaptive Reranking.

GPU-utilized Composite Quantization for Large-scale High-dimension ANN Search

PQBF: I/O-Efficient Approximate Nearest Neighbor Search by Product Quantization.

Improving Bilayer Product Quantization for Billion-Scale Approximate Nearest Neighbors in High Dimensions

Efficient Large-Scale Approximate Nearest Neighbor Search on OpenCL FPGA

Multi-Pqtable For Approximate Nearest-Neighbor Search

Scalable Image Retrieval by Sparse Product Quantization

Hash Learning with Variable Quantization for Large-scale Retrieval

Local Deep Learning Quantization for Approximate Nearest Neighbor Search

Indexing Large Visual Vocabulary by Randomized Dimensions Hashing for High Quantization Accuracy: Improving the Object Retrieval Quality

Flexible Product Quantization for Fast Approximate Nearest Neighbor Search.

Routing-Guided Learned Product Quantization for Graph-Based Approximate Nearest Neighbor Search

Projected Residual Vector Quantization for ANN Search

DiskANN++: Efficient Page-based Search over Isomorphic Mapped Graph Index using Query-sensitivity Entry Vertex

Practical and Asymptotically Optimal Quantization of High-Dimensional Vectors in Euclidean Space for Approximate Nearest Neighbor Search

Unleashing the Full Potential of Product Quantization for Large-Scale Image Retrieval.

Dynamic programming based optimized product quantization for approximate nearest neighbor search.

RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search