Abstract:The set similarity join finds all pairs of similar sets from two collections of sets. It has many real world applications, such as personalized recommendation and community mining. In this paper, we study the problem of computing the similarity join in a dynamic context, where the sets are updated dynamically. This, however, is inefficient with the state-of-the-art join methods, because they usually assume that data collections are static and have to compute the join result from scratch whenever a set is updated. To address this issue, we propose ${{\sf ALJoin}}$ , an adaptive filtering approach that computes the join result incrementally based on the update logs. We first investigate the effect of set updates on the similarity values, and on this basis we propose to build a neighborhood index for each set. The neighborhood index of a specific set consists of any other sets that can be transformed into its similar sets within a threshold number of update operations. ${{\sf ALJoin}}$ then uses this index to effectively identify both similar and dissimilar set pairs based on their update logs. To efficiently build the neighborhood index, we devise several filtering techniques and propose a “lazy-forward” method to reduce the computational cost. In addition, to improve the efficiency on varying workloads, we propose an analytical cost model, and design an online algorithm with performance guarantees to dynamically consolidate the update logs and adapt the neighborhood indexes. We evaluated our method using four real-world datasets. Experimental results show that our approach outperforms existing methods by up to $3.7\times$ .

An Efficient Partition Based Method for Exact Set Similarity Joins

Improved LSH-driven String Similarity Join Filtering-Verification Framework

Set Similarity Join Using Partition Index

A Partition-Based Method for String Similarity Joins with Edit-Distance Constraints

PASS-JOIN: A Partition-based Method for Similarity Joins

Overlap Set Similarity Joins with Theoretical Guarantees.

Dynamic Set Similarity Join: an Update Log Based Approach

Efficient Parallel Partition-Based Algorithms for Similarity Search and Join with Edit Distance Constraints

An Efficient Framework for Exact Set Similarity Search Using Tree Structure Indexes.

An Efficient MapReduce Algorithm for Similarity Join in Metric Spaces

Trie-join: a Trie-Based Method for Efficient String Similarity Joins

K-Join: Knowledge-Aware Similarity Join

Efficient Similarity Join and Search on Multi-Attribute Data

Improving Distributed Similarity Join in Metric Space with Error-bounded Sampling

Massjoin: A Mapreduce-Based Method for Scalable String Similarity Joins

Fast-join: an Efficient Method for Fuzzy Token Matching Based String Similarity Join

Similarity Query Processing for Probabilistic Sets

String Similarity Joins

Metric Similarity Joins Using MapReduce

Metric Similarity Joins Using Mapreduce (Extended Abstract)

Hash(Ed)-Join: Approximate String Similarity Join With Hashing