Abstract:Gene sequence search is a fundamental operation in computational genomics. Due to the petabyte scale of genome archives, most gene search systems now use hashing-based data structures such as Bloom Filters (BF). The state-of-the-art systems such as Compact bit-slicing signature index (COBS) and Repeated And Merged Bloom filters (RAMBO) use BF with Random Hash (RH) functions for gene representation and identification. The standard recipe is to cast the gene search problem as a sequence of membership problems testing if each subsequent gene substring (called kmer) of Q is present in the set of kmers of the entire gene database D. We observe that RH functions, which are crucial to the memory and the computational advantage of BF, are also detrimental to the system performance of gene-search systems. While subsequent kmers being queried are likely very similar, RH, oblivious to any similarity, uniformly distributes the kmers to different parts of potentially large BF, thus triggering excessive cache misses and causing system slowdown. We propose a novel hash function called the Identity with Locality (IDL) hash family, which co-locates the keys close in input space without causing collisions. This approach ensures both cache locality and key preservation. IDL functions can be a drop-in replacement for RH functions and help improve the performance of information retrieval systems. We give a simple but practical construction of IDL function families and show that replacing the RH with IDL functions reduces cache misses by a factor of 5x, thus improving query and indexing times of SOTA methods such as COBS and RAMBO by factors up to 2x without compromising their quality. We also provide a theoretical analysis of the false positive rate of BF with IDL functions. Our hash function is the first study that bridges Locality Sensitive Hash (LSH) and RH to obtain cache efficiency.

Hashing with Linear Probing and Referential Integrity

A New Linkless Hierarchical Structure Based on Perfect Hashing

Lock-free Dynamic Hash Tables with Open Addressing.

On the Relationship Between Several Variants of the Linear Hashing Conjecture

Linear Hashing with $\ell_\infty$ guarantees and two-sided Kakeya bounds

Defending Hash Tables from Subterfuge with Depth Charge

Extended Probe Method for Linkage Discovery over High-Cardinality Alphabets.

Iceberg Hashing: Optimizing Many Hash-Table Criteria at Once

Shifting Hash Table: An Efficient Hash Table With Delicate Summary

A New Hashing Function: Statistical Behaviour and Algorithm

Identifying Valuable Pointers in Heap Data

Stochastic Probing with Increasing Precision

DHash: Dynamic Hash Tables With Non-Blocking Regular Operations

A New Approach to Analyzing Robin Hood Hashing

Query Range Sensitive Probability Guided Multi-probe Locality Sensitive Hashing

Modular Verification of Linearizability with Non-Fixed Linearization Points

IDentity with Locality: An ideal hash for gene sequence search

Skip Hash: A Fast Ordered Map Via Software Transactional Memory

DHash: Enabling Dynamic and Efficient Hash Tables

Cache-Aware Lock-Free Concurrent Hash Tries

A Hash Table Without Hash Functions, and How to Get the Most Out of Your Random Bits