Abstract:Approximate K nearest neighbor (AKNN) search in high-dimensional Euclidean space is a fundamental problem with widespread applications. Vector quantization which maps vectors to discrete quantized code, can significantly reduce the space cost of AKNN search while also accelerating the AKNN search speed. The exclusive use of vector quantization without precise vectors leads to a substantial decline in search accuracy. Recent research RaBitQ addresses this issue by using geometry relation to enhance quantization accuracy and employing error bound for distance correction with precise vector. However, this method requires that the quantization bit must be equal to the vector dimension resulting in a fixed compression ratio which limits its efficiency and flexibility. In this paper, we propose a new and efficient method MRQ to address this drawback. MRQ leverage leverages data distribution to achieve better distance correction and a higher vector compression ratio. MRQ reduces the query latency based on a highly efficient distance computation and correction scheme. Our results demonstrate that MRQ significantly outperforms state-of-the-art AKNN search methods based on graph or vector quantization, achieving up to a 3x efficiency speed-up with only 1/3 length of quantized code while maintaining the same accuracy.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the problems of efficiency and accuracy in approximate K - nearest neighbor (AKNN) search in high - dimensional spaces. Specifically, the researchers focus on the following points: 1. **Improving the compression ratio and flexibility**: - Existing quantization methods such as RaBitQ require the number of quantization bits to be equal to the vector dimension, resulting in a fixed compression ratio (e.g., 32 times), which limits their efficiency and flexibility. - The paper proposes a new method MRQ (Minimization Residual Quantization), which achieves a higher compression ratio and more flexible quantization bit lengths by leveraging data distribution characteristics. 2. **Enhancing retrieval accuracy and speed**: - Vector quantization can significantly reduce storage costs and accelerate AKNN search, but it will lead to a significant drop in retrieval accuracy. - MRQ achieves a up to 3 - fold improvement in search efficiency while maintaining high retrieval accuracy and only requires 1/3 - length quantization codes through a multi - stage distance correction framework. 3. **Optimizing the distance correction scheme**: - Existing distance correction methods such as re - ranking and error - bound - based methods have problems such as low efficiency and the need for a large number of exact distance calculations. - MRQ designs a multi - stage distance correction method by analyzing data distribution and variance characteristics after projection, reducing redundant operations and improving execution efficiency in the hardware environment. ### Main contributions - **Data analysis**: Through the analysis of existing vector data sets, it is found that the high - dimensional vector variances after PCA rotation show a long - tail distribution, indicating that most dimensions contribute less to the variance, so distance correction can be carried out with fewer dimensions. - **New distance correction method**: Based on the observation of data distribution, a multi - stage distance correction method is designed, including binary quantization, quadratic correction of projection vectors, and final exact distance calculation, ensuring efficient and accurate distance correction in the hardware environment. - **Efficient implementation**: The method is applied to IVF - based AKNN indexes. By optimizing the memory layout through phased distance correction, the cache hit rate is increased, and the IVF centroids are further approximated to enhance the index search efficiency. - **Experimental verification**: Extensive experiments were carried out on multiple real - data sets. Compared with existing state - of - the - art methods such as HNSW and RabitQ, MRQ performs excellently on multiple data sets, achieving a significant improvement in search efficiency while maintaining the same retrieval accuracy. In summary, this paper mainly solves the problems of efficiency and accuracy in AKNN search in high - dimensional spaces. By introducing the MRQ method, it achieves faster and more accurate retrieval while improving the compression ratio and flexibility.

Fast High-dimensional Approximate Nearest Neighbor Search with Efficient Index Time and Space

Accelerating Exact Nearest Neighbor Search in High Dimensional Euclidean Space Via Block Vectors

Practical and Asymptotically Optimal Quantization of High-Dimensional Vectors in Euclidean Space for Approximate Nearest Neighbor Search

Improved Residual Vector Quantization for High-dimensional Approximate Nearest Neighbor Search

RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search

Constrained All-k-Nearest-Neighbor Search

Fast Additive Quantization for Vector Compression in Nearest Neighbor Search.

Efficient Reverse $k$ Approximate Nearest Neighbor Search over High-Dimensional Vectors

Effective and General Distance Computation for Approximate Nearest Neighbor Search

Approximate Nearest Neighbor Search on High Dimensional Data — Experiments, Analyses, and Improvement

Derived Codebooks for High-Accuracy Nearest Neighbor Search

Local Deep Learning Quantization for Approximate Nearest Neighbor Search

Equal-Average Equal-Variance Equal-Norm Nearest Neighbor Search Algorithm for Vector Quantization

Distributed Adaptive Binary Quantization for Fast Nearest Neighbor Search.

Transformed Residual Quantization for Approximate Nearest Neighbor Search

Double-Bit Quantization and Index Hashing for Nearest Neighbor Search

Fast Adaptive Similarity Search through Variance-Aware Quantization

Fast Image Search Using Vector Quantization

Accelerated Nearest Neighbor Search with Quick ADC

Equal-Average Equal-Variance Nearest Neighbor Search Algorithm Based on Hadamard Transform

Fast Nearest Neighbor Search in the Hamming Space.