Abstract:Similar substring matching, as an essential operation in applications including read mapping and text retrieval, has attracted significant attention in the research community. In this paper, we study the problem of similar substring matching with edit distance constraints. Existing methods generally utilize a filtering-and-verification framework to solve this problem – a filtering procedure is employed to reduce the searching space before going to a computationally expensive verification step, and the efficiency depends critically on balancing the cost of filtering and verification. The common filtering paradigm is based on the principle of Pigeonhole stating that a matching result must exactly match at least a certain number of substrings from the query, where the substrings act as a filter. However, the polynomial growth of filters caused by enlarging the number of substrings in filters, leading to the cost of filtering and verification is not well-balanced for the existing methods. To this end, we propose a novel filtering paradigm hierarchical filtering , aiming at achieving a fine-grained balance on the cost of filtering and verification. Unlike using a fixed number of substrings in a filter, our method allows the filters contain a different number of substrings that avoids the polynomial growth of filters. The filters are picked in accord with a scoring metric. We devise a tree-based filtering framework for hierarchical filtering. Also, the cost of filtering and verification is further reduced by eliminating the duplication of filters. Extensive experiments have been conducted on four real-world datasets by comparing to state-of-the-art methods Hobbes3, BWA, and BLAST, etc. The results show that our method outperforms the competing methods under a wide range of parameter settings.

PartSS: an Efficient Partition-Based Filtering for Edit Distance Constraints

Improved LSH-driven String Similarity Join Filtering-Verification Framework

FrepJoin: an Efficient Partition-Based Algorithm for Edit Similarity Join

A Partition-Based Method for String Similarity Joins with Edit-Distance Constraints

PASS-JOIN: A Partition-based Method for Similarity Joins

A Pivotal Prefix Based Filtering Algorithm for String Similarity Search

Efficient String Similarity Search: A Cross Pivotal Based Approach.

A unified framework for string similarity search with edit-distance constraint

Hierarchical filtering: improving similar substring matching under edit distance

VChunkJoin: an Efficient Algorithm for Edit Similarity Joins

Efficient and Scalable Processing of String Similarity Join

Efficient Parallel Partition-Based Algorithms for Similarity Search and Join with Edit Distance Constraints

Pass-Join-K: Similarity Join Method Based on Multi-Match Partition

Efficiently Supporting Edit Distance Based String Similarity Search Using B<formula Formulatype="inline"><tex Notation="TeX"> $^+$</tex><mathgraphic Graphicformat="gif" Fileref="lu-Ieq1-2309131.gif"/></formula>-trees

Top-k String Similarity Search with Edit-Distance Constraints

Set Similarity Join Using Partition Index

Shifted Hamming distance: a fast and accurate SIMD-friendly filter to accelerate alignment verification in read mapping

GFilter: A General Gram Filter for String Similarity Search.

Efficient EMD-based Similarity Search Via Batch Pruning and Incremental Computation (extended Abstract)

An Efficient Partition Based Method for Exact Set Similarity Joins

Large-Scale Similarity Join With Edit-Distance Constraints