Abstract:With the explosive growth of high-dimensional data, approximate methods emerge as promising solutions for nearest neighbor search. Among alternatives, quantization methods have gained attention due to the fast query responses and the low encoding and storage costs. Quantization methods decompose data dimensions into non-overlapping subspaces and encode data using a different dictionary per subspace. The state-of-the-art approach assigns dictionary sizes uniformly across subspaces while attempting to balance the relative importance of subspaces. Unfortunately, a uniform balance is not always achievable and may lead to unsatisfactory performance. Similarly, hardware-accelerated quantization methods may sacrifice accuracy to speed up the query execution. We propose a Variance-Aware Quantization (VAQ) method to encode data by intelligently adapting dictionary sizes to subspaces to alleviate these significant drawbacks. VAQ exploits intrinsic dimensionality reduction properties to derive the subspaces and only partially balances the importance of subspaces. Then, VAQ solves a constrained optimization problem to assign dictionary sizes proportionally to the importance of each subspace. In addition, VAQ accelerates the query execution by skipping data and subspaces through a hardware-oblivious algorithmic solution. To demonstrate the robustness of VAQ, we perform an extensive evaluation against quantization, hashing, and indexing methods using five large-scale benchmarking datasets. VAQ significantly outperforms the strongest hashing and quantization methods in accuracy while achieving up to 5× speedup. Compared to the fastest but less accurate hardware-accelerated method, VAQ achieves a speedup@recall performance up to 14×. Importantly, a rigorous statistical comparison using over one hundred datasets reveals that VAQ significantly outperforms rival methods even with a half budget. Notably, VAQ's simple data skipping solution achieves competitive or better performance against index-based methods, highlighting the need for new indices for quantization methods.

Exclusive visual descriptor quantization

Equal-Average Equal-Variance Equal-Norm Nearest Neighbor Search Algorithm for Vector Quantization

Fast Visual Word Quantization Via Spatial Neighborhood Boosting

Fast Norm-Ordered Nearest-Neighbor Codeword Search Algorithms For Image Vector Quantization

Equal-Average Equal-Variance Nearest Neighbor Search Algorithm Based on Hadamard Transform

An efficient encoding algorithm for vector quantization based on subvector technique.

VQ Image Coding Using Sub-Vector Techniques.

&Lt;title>image Coding Using Variable-Rate Classified Side-Match Vector Quantization</title>

Edge and Contrast Classified K-means Algorithm for Image Vector Quantizer Design.

Fast Adaptive Similarity Search through Variance-Aware Quantization

Fast Search In Large-Scale Image Database Using Vector Quantization

Fast Image Search Using Vector Quantization

Deep Visual-Semantic Quantization For Efficient Image Retrieval

Transformed Residual Quantization for Approximate Nearest Neighbor Search

SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization

Hash Learning with Variable Quantization for Large-scale Retrieval

Visual Words Refining Exploiting Spatial Co-Occurrence Table

Local Deep Learning Quantization for Approximate Nearest Neighbor Search

Fast Search Algorithm for Vector Quantization

Resizing codebook of vector quantization without retraining

Fast High-dimensional Approximate Nearest Neighbor Search with Efficient Index Time and Space