Abstract:A distance search in Hamming space finds binary vectors whose Hamming distances are no more than a threshold from a query vector. It is a fundamental problem in many applications, such as image retrieval, near-duplicate Web page detection, and scientific databases. State-of-the-art approaches to Hamming distance search are mainly based on the pigeonhole principle to generate a set of candidates and then verify them. We observe that the constraint by the pigeonhole principle is not always tight and may bring about unnecessary candidates. We also observe that the distribution in real data is often skewed, but most existing solutions adopt a simple equi-width partitioning and allocate the same threshold to all the parts, hence failing to exploit the data skewness to optimize query processing. In this paper, we propose a new form of the pigeonhole principle which allows variable partitioning and threshold allocation. Based on the new principle, we develop a tight constraint of candidates and devise cost-aware methods for partitioning and threshold allocation to optimize query processing. In addition, we extend our methods to answer Hamming distance join queries. We also discuss the application of the pigeonhole principle in set similarity search, a problem that can be converted to Hamming distance search equivalently. Our evaluation on datasets with various data distributions shows the robustness of our solution and its superior query processing performance to the state-of-the-art methods.

Hamming Distance Based Approximate Similarity Text Search Algorithm

An efficient method for time series similarity search using binary code representation and hamming distance

Efficient Processing of Hamming-Distance-Based Similarity-Search Queries over MapReduce.

Least Square Regularized Spectral Hashing for Similarity Search

Generalizing the Pigeonhole Principle for Similarity Search in Hamming Space

Hashing for Similarity Search: A Survey.

Rank Hash Similarity for Fast Similarity Search

Semi-supervised Spectral Hashing for Fast Similarity Search

Semi-supervised SimHash for Efficient Document Similarity Search

An Improved Hash Function Inspired by the Fly Hashing for Near Duplicate Detections

Supervised Discrete Hashing for Hamming Space Retrieval

Application of the maintenance text data of transformers based on SimHash and Hamming distance algorithm

Hash(Ed)-Join: Approximate String Similarity Join With Hashing

Discrete Spectral Hashing for Efficient Similarity Retrieval.

Effective Indices for Efficient Approximate String Search and Similarity Join

Sparse Hashing for Fast Multimedia Search.

Sparse Semantic Hashing for Efficient Large Scale Similarity Search

Local Similarity Preserved Hashing Learning Via Markov Graph for Efficient Similarity Search

Scalable Similarity Search with Topology Preserving Hashing

Fast kNN Search in Weighted Hamming Space With Multiple Tables

A Vector Space Projection HITS Algorithm Based on Similarity Value