Abstract:With the explosive growth of high-dimensional data, approximate methods emerge as promising solutions for nearest neighbor search. Among alternatives, quantization methods have gained attention due to the fast query responses and the low encoding and storage costs. Quantization methods decompose data dimensions into non-overlapping subspaces and encode data using a different dictionary per subspace. The state-of-the-art approach assigns dictionary sizes uniformly across subspaces while attempting to balance the relative importance of subspaces. Unfortunately, a uniform balance is not always achievable and may lead to unsatisfactory performance. Similarly, hardware-accelerated quantization methods may sacrifice accuracy to speed up the query execution. We propose a Variance-Aware Quantization (VAQ) method to encode data by intelligently adapting dictionary sizes to subspaces to alleviate these significant drawbacks. VAQ exploits intrinsic dimensionality reduction properties to derive the subspaces and only partially balances the importance of subspaces. Then, VAQ solves a constrained optimization problem to assign dictionary sizes proportionally to the importance of each subspace. In addition, VAQ accelerates the query execution by skipping data and subspaces through a hardware-oblivious algorithmic solution. To demonstrate the robustness of VAQ, we perform an extensive evaluation against quantization, hashing, and indexing methods using five large-scale benchmarking datasets. VAQ significantly outperforms the strongest hashing and quantization methods in accuracy while achieving up to 5× speedup. Compared to the fastest but less accurate hardware-accelerated method, VAQ achieves a speedup@recall performance up to 14×. Importantly, a rigorous statistical comparison using over one hundred datasets reveals that VAQ significantly outperforms rival methods even with a half budget. Notably, VAQ's simple data skipping solution achieves competitive or better performance against index-based methods, highlighting the need for new indices for quantization methods.

Fast Search In Large-Scale Image Database Using Vector Quantization

Fast Image Search Using Vector Quantization

Large-scale Image Retrieval Based on Boosting Iterative Quantization Hashing with Query-Adaptive Reranking.

Color Image Retrieval Utilizing Extended Fast Vq Codeword Search Technique And Vector Composition-Based Feedback

Hierarchical Indexing Scheme for Fast Search in Large-Scale Image Database

Hierarchical indexing scheme for fast search in a large-scale image database

Real-Time Image Semantic Retrieval Based on VQ

Efficient Indexing for Large Scale Visual Search

Scalar Quantization for Large Scale Image Search

Fast Image Retrieval Method Based on Lattice Vector Quantization and Inverted File

Fast Norm-Ordered Nearest-Neighbor Codeword Search Algorithms For Image Vector Quantization

Fast Adaptive Similarity Search through Variance-Aware Quantization

Sub-Selective Quantization for Large-Scale Image Search

Deep Visual-Semantic Quantization For Efficient Image Retrieval

Vector and Line Quantization for Billion-scale Similarity Search on GPUs

A Spatial/Frequency Hybrid Vector Quantizer Based on a Classification in the DCT Domain

Colour Image Retrieval Based on DCT-domain Vector Quantisation Index Histograms

Image Retrieval Using VQ Index Histograms in Classified DCT Domain.

VQ Image Coding Using Sub-Vector Techniques.

Image Retrieval Method Based On Fast Equal-Average K Nearest Neighbor Search And Query-Feature-Vector-Recomposition Feedback

Large scale partial-duplicate image retrieval with bi-space quantization and geometric consistency