Abstract:The set similarity join finds all pairs of similar sets from two collections of sets. It has many real world applications, such as personalized recommendation and community mining. In this paper, we study the problem of computing the similarity join in a dynamic context, where the sets are updated dynamically. This, however, is inefficient with the state-of-the-art join methods, because they usually assume that data collections are static and have to compute the join result from scratch whenever a set is updated. To address this issue, we propose ${{\sf ALJoin}}$ , an adaptive filtering approach that computes the join result incrementally based on the update logs. We first investigate the effect of set updates on the similarity values, and on this basis we propose to build a neighborhood index for each set. The neighborhood index of a specific set consists of any other sets that can be transformed into its similar sets within a threshold number of update operations. ${{\sf ALJoin}}$ then uses this index to effectively identify both similar and dissimilar set pairs based on their update logs. To efficiently build the neighborhood index, we devise several filtering techniques and propose a “lazy-forward” method to reduce the computational cost. In addition, to improve the efficiency on varying workloads, we propose an analytical cost model, and design an online algorithm with performance guarantees to dynamically consolidate the update logs and adapt the neighborhood indexes. We evaluated our method using four real-world datasets. Experimental results show that our approach outperforms existing methods by up to $3.7\times$ .

Efficient Similarity Join and Search on Multi-Attribute Data

Improved LSH-driven String Similarity Join Filtering-Verification Framework

Enhancing Multi-Attribute Similarity Join using Reduced and Adaptive Index Trees

Can we beat the prefix filtering?: an adaptive framework for similarity join and search.

Reserch of Entity Matching Based on Multiple Heterogenous Data

An Efficient Framework for Exact Set Similarity Search Using Tree Structure Indexes.

Efficient Similarity Search for Tree-Structured Data

Effective Indices for Efficient Approximate String Search and Similarity Join

An Efficient MapReduce Algorithm for Similarity Join in Metric Spaces

An Efficient Partition Based Method for Exact Set Similarity Joins

Efficient Parallel Partition-Based Algorithms for Similarity Search and Join with Edit Distance Constraints

Trie-join: a Trie-Based Method for Efficient String Similarity Joins

Hash(Ed)-Join: Approximate String Similarity Join With Hashing

DESIRE: An Efficient Dynamic Cluster-based Forest Indexing for Similarity Search in Multi-Metric Spaces

Joinable Search over Multi-source Spatial Datasets: Overlap, Coverage, and Efficiency

K-Join: Knowledge-Aware Similarity Join

PASS-JOIN: A Partition-based Method for Similarity Joins

Efficient Metric Indexing for Similarity Search

Fast Comparative Analysis of Merge Trees Using Locality Sensitive Hashing

Dynamic Set Similarity Join: an Update Log Based Approach

Top-k Similarity Join in Heterogeneous Information Networks