Abstract:In this paper, we study the problem of set containment join. Given two collections \(\mathcal {R}\) and \(\mathcal {S}\) of records, the set containment join \(\mathcal {R} \bowtie _\subseteq \mathcal {S}\) retrieves all record pairs \(\{(r,s)\} \in \mathcal {R}\times \mathcal {S}\) such that \(r \subseteq s\). This problem has been extensively studied in the literature and has many important applications in commercial and scientific fields. Recent research focuses on the in-memory set containment join algorithms, and several techniques have been developed following intersection-oriented or union-oriented computing paradigms. Nevertheless, we observe that two computing paradigms have their limits due to the nature of the intersection and union operators. Particularly, intersection-oriented method relies on the intersection of the relevant inverted lists built on the elements of \(\mathcal {S}\). A nice property of the intersection-oriented method is that the join computation is verification free. However, the number of records explored during the join process may be large because there are multiple replicas for each record in \(\mathcal {S}\). On the other hand, the union-oriented method generates a signature for each record in \(\mathcal {R}\) and the candidate pairs are obtained by the union of the inverted lists of the relevant signatures. The candidate size of the union-oriented method is usually small because each record contributes only one replica in the index. Unfortunately, union-oriented method needs to verify the candidate pairs, which may be cost expensive especially when the join result size is large. As a matter of fact, the state-of-the-art union-oriented solution is not competitive compared to the intersection-oriented ones. In this paper, we propose a new union-oriented method, namely TT-Join, which not only enhances the advantage of the previous union-oriented methods but also integrates the goodness of intersection-oriented methods by imposing a variant of prefix tree structure. We conduct extensive experiments on 20 real-life datasets and synthetic datasets by comparing our method with 7 existing methods. The experiment results demonstrate that TT-Join significantly outperforms the existing algorithms on most of the datasets and can achieve up to two orders of magnitude speedup. Furthermore, to support large scale of datasets, we extend our techniques to distributed systems on top of MapReduce framework. With the help of careful designed load-aware distribution mechanisms, our distributed join algorithm can achieve up to an order of magnitude speedup than the baselines methods.

Solving k-Set Chain Range-Join With Forest

Fast Combination of Scheduling Chains under Resource and Time Constraints

Solving k-Set Agreement with Stable Skeleton Graphs

Tight Approximation for Variants of Directed Steiner Tree Via State-Tree Decomposition and Linear Programming Rounding.

Overlap Set Similarity Joins with Theoretical Guarantees.

Intersecting families with full difference sets

Exact and Parameterized Algorithms for Restricted Subset Feedback Vertex Set in Chordal Graphs

A faster algorithm for the $k$-forest problem: breaking the $O_k(n^{3/2})$ complexity barrier

A Hitting Set Relaxation for $k$-Server and an Extension to Time-Windows

Efficient Set Containment Join

Counting spanning trees in a complete bipartite graph which contain a given spanning forest

Approximation Algorithms for Somek-Source Shortest Paths Spanning Tree Problems

Extremal polyphenyl chains concerning k-matchings and k-independent sets.

Minimum Chain Cover in Almost Linear Time

Densest K-Subgraph Approximation On Intersection Graphs

Pseudo-polynomial Algorithm for the K- ( N1, 1,……, 1 ) - Chains Problem with Lower Bound Delays

Non-uniform Cross-intersecting Families

A note on non-empty cross-intersecting families

Intersecting families of sets are typically trivial

FreshJoin: an Efficient and Adaptive Algorithm for Set Containment Join

An efficient processing of a chain join with the minimum communication cost in distributed database systems