Abstract:We study the exact set similarity join problem, which, given two collections of sets, finds out all the similar set pairs from the collections. Existing methods generally utilize the prefix filter based framework. They generate a prefix for each set and prune all the pairs whose prefixes are disjoint. However the pruning power is limited, because if two dissimilar sets share a common element in their prefixes, they cannot be pruned. To address this problem, we propose a partition-based framework. We design a partition scheme to partition the sets into several subsets and guarantee that two sets are similar only if they share a common subset. To improve the pruning power, we propose a mixture of the subsets and their 1-deletion neighborhoods (the subset of a set by eliminating one element). As there are multiple allocation strategies to generate the mixture, we evaluate different allocations and design a dynamic-programming algorithm to select the optimal one. However the time complexity of generating the optimal one is O(s3) for a set with size s. To speed up the allocation selection, we develop a greedy algorithm with an approximation ratio of 2. To further reduce the complexity, we design an adaptive grouping mechanism, and the two techniques can reduce the complexity to O(s log s). Experimental results on three real-world datasets show our method achieves high performance and outperforms state-of-the-art methods by 2-5 times.

Efficient Similarity Joins for Near-Duplicate Detection

Improved LSH-driven String Similarity Join Filtering-Verification Framework

An Integrated Approach for Detecting Approximate Duplicate Records

Efficient Near-Duplicate Detection for Q&A Forum.

Fast And Robust Detection Of Near-Duplicates In Web Video Database

An Efficient Partition Based Method for Exact Set Similarity Joins

Efficient Similarity Join and Search on Multi-Attribute Data

Efficient Partial-Duplicate Detection Based on Sequence Matching

Efficient near-duplicate image detection by learning from examples

Detecting Nearly Duplicated Records in Location Datasets.

Efficient Parallel Partition-Based Algorithms for Similarity Search and Join with Edit Distance Constraints

An Efficient MapReduce Algorithm for Similarity Join in Metric Spaces

Efficient and Accurate SimRank-Based Similarity Joins: Experiments, Analysis, and Improvement

Query Oriented Subspace Shifting for Near-Duplicate Image Detection

Expressing And Optimizing Similarity-Based Queries In Sql

Hash(Ed)-Join: Approximate String Similarity Join With Hashing

Efficient EMD-based Similarity Search Via Batch Pruning and Incremental Computation (extended Abstract)

Can we beat the prefix filtering?: an adaptive framework for similarity join and search.

A Pivotal Prefix Based Filtering Algorithm for String Similarity Search

Effective Indices for Efficient Approximate String Search and Similarity Join

Efficient index-based KNN join processing for high-dimensional data