Metric Similarity Joins Using MapReduce

ChenGang,YangKeyu,ChenLu,GaoYunjun,ZhengBaihua,ChenChun

IF: 9.235

2017-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Given two object sets Q and O, a metric similarity join finds similar object pairs according to a certain criterion. This operation has a wide variety of applications in data cleaning and data mini...

What problem does this paper attempt to address?

Metric Similarity Joins Using Mapreduce (Extended Abstract)

Gang Chen,Keyu Yang,Lu Chen,Yunjun Gao,Baihua Zheng,Chun Chen

DOI: https://doi.org/10.1109/ICDE.2018.00251

2018-01-01

Abstract:Given two object sets Q and O, a metric similarity join finds similar object pairs according to a certain criterion. This operator has a wide range of applications in data cleaning, data mining, etc. In this paper, we employ a popular distributed framework, namely, MapReduce, to support scalable metric similarity joins. To ensure load balancing, we present two sampling based partition methods, i.e., clustering based partition method and KD-tree based partition method. To avoid unnecessary object pair evaluation, we propose a framework that maps the two involved object sets in order, where plane sweeping and pivot based filtering techniques are utilized for pruning. Extensive experiments confirm that our solution outperforms significantly existing state-of-the-art competitors.
Metric Similarity Joins Using MapReduce

Gang Chen,Keyu Yang,Lu Chen,Yunjun Gao,Baihua Zheng,Chun Chen

DOI: https://doi.org/10.1109/tkde.2016.2631599

IF: 9.235

2017-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Given two object sets Q and O, a metric similarity join finds similar object pairs according to a certain criterion. This operation has a wide variety of applications in data cleaning and data mining, to name but a few. However, the rapidly growing volume of data nowadays challenges traditional metric similarity join methods, and thus, a distributed method is required. In this paper, we adopt a popular distributed framework, namely, MapReduce, to support scalable metric similarity joins. To ensure the load balancing, we present two sampling based partition methods. One utilizes the pivot and the space-filling curve mappings to cluster the data into one-dimensional space, and then selects high quality centroids to enable equal-sized partitions. The other uses the KD-tree partitioning technique to equally divide the data after the pivot mapping. To avoid unnecessary object pair evaluation, we propose a framework that maps the two involved object sets in order, where the range-object filtering, the double-pivot filtering, the pivot filtering, and the plane sweeping techniques are utilized for pruning. Extensive experiments with both real and synthetic data sets demonstrate that our solutions outperform significantly existing state-of-the-art competitors.
An Efficient MapReduce Algorithm for Similarity Join in Metric Spaces

Wen Liu,Yanming Shen,Peng Wang

DOI: https://doi.org/10.1007/s11227-016-1651-9

2016-01-01

Abstract:Given a massive set of records, similarity join is to find pairs of records with similarity score greater than a threshold. In this paper, we address the problem of scaling up similarity join for general metric distance functions using MapReduce. First, we propose a novel index structure, Similarity Join Tree (SJT), which partitions data based on the underlying data distribution, and distributes similar records to the same group. Different from existing approaches, SJT can prune a large number of comparisons within reduce tasks by utilizing the by-product results generated in partitioning data. Then, to avoid the straggler reduce tasks, we design a graph partition algorithm by extending the well known Fiduccia–Mattheyses algorithm which can ensure load balancing while minimizing communication cost and redundancy in all reduce tasks. Experimental results using real data sets show that our approach is more effective and scalable compared to state-of-the-art algorithms.
Scalable Metric Similarity Join Using MapReduce

Jiacheng Wu,Yong Zhang,Jin Wang,Chunbin Lin,Yingjia Fu,Chunxiao Xing

DOI: https://doi.org/10.1109/icde.2019.00167

2019-01-01

Abstract:Given two collections of objects, metric similarity join finds all similar pairs of objects according to a particular distance function in metric space. There is an increasing demand to provide a scalable similarity join algorithm which can support efficient query and analytical services in the era of Big Data. In this paper, we propose SMS-Join, a parallel framework to support similarity join in metric space based on the MapReduce paradigm. The overall workflow of SMS-Join is that it first finds some records as pivots in the preprocessing phase and then splits the data into partitions based on them with a map job. Finally the join results are obtained via a reduce job. To ensure load balancing between the partitions, we devise a light-weighted sampling technique to obtain high quality samples while maintaining the high performance. To reduce the partition cost, we develop an iterative partition strategy in the map phase. We implement our framework upon Apache Spark platform and conduct extensive experiments on four real world datasets. The results show that our method significantly outperforms state-of-the-art methods.
Improve Semantic Web Services Discovery Through Similarity Search in Metric Space

Minghui Wu,Fanwei Zhu,Jia Lv,Tao Jiang,Jing Ying

DOI: https://doi.org/10.1109/tase.2009.50

2009-01-01

Abstract:Most current semantic web services (SWS) discovery approaches focus on the matchmaking of services in a specific description language while in practical application the advertised services are often heterogeneous and distributed. This paper proposes a metric space approach to resolve this problem in which all heterogeneous web services are modeled as metric objects regardless of concrete description languages, and thereby the discovery problem can be treated as similarity search in metric space with a uniform criterion. In the matchmaking process, both the functional semantics and non-functional semantics of the web services are integrated as selection conditions for similarity query. And two types of similarity queries: range query and an improved nearest neighbor query are combined to produce a sorted result set.
Efficient Top-K Similarity Join Processing over Multi-Valued Objects

Wenjie Zhang,Liming Zhan,Ying Zhang,Muhammad Aamir Cheema,Xuemin Lin

DOI: https://doi.org/10.1007/s11280-012-0201-5

2013-01-01

World Wide Web

Abstract:The top-k similarity joins have been extensively studied and used in a wide spectrum of applications such as information retrieval, decision making, spatial data analysis and data mining. Given two sets of objects $\mathcal U$ and $\mathcal V$, a top-k similarity join returns k pairs of most similar objects from $\mathcal U \times \mathcal V$. In the conventional model of top-k similarity join processing, an object is usually regarded as a point in a multi-dimensional space and the similarity is measured by some simple distance metrics like Euclidean distance. However, in many applications an object may be described by multiple values (instances) and the conventional model is not applicable since it does not address the distributions of object instances. In this paper, we study top-k similarity join over multi-valued objects. We apply two types of quantile based distance measures, ϕ-quantile distance and ϕ-quantile group-base distance, to explore the relative instance distribution among the multiple instances of objects. Efficient and effective techniques to process top-k similarity joins over multi-valued objects are developed following a filtering-refinement framework. Novel distance, statistic and weight based pruning techniques are proposed. Comprehensive experiments on both real and synthetic datasets demonstrate the efficiency and effectiveness of our techniques.
Improving Distributed Similarity Join in Metric Space with Error-bounded Sampling

Jiacheng Wu,Yong Zhang,Jin Wang,Chunbin Lin,Yingjia Fu,Chunxiao Xing

DOI: https://doi.org/10.48550/arxiv.1905.05981

2019-01-01

Abstract:Given two sets of objects, metric similarity join finds all similar pairs of objects according to a particular distance function in metric space. There is an increasing demand to provide a scalable similarity join framework which can support efficient query and analytical services in the era of Big Data. The existing distributed metric similarity join algorithms adopt random sampling techniques to produce pivots and utilize holistic partitioning methods based on the generated pivots to partition data, which results in data skew problem since both the generated pivots and the partition strategies have no quality guarantees. To address the limitation, we propose SP-Join, an end-to-end framework to support distributed similarity join in metric space based on the MapReduce paradigm, which (i) employs an estimation-based stratified sampling method to produce pivots with quality guarantees for any sample size, and (ii) devises an effective cost model as the guideline to split the whole datasets into partition in map and reduce phases according to the sampled pivots. Although obtaining an optimal set of partitions is proven to be NP-Hard, SP-Join adopts efficient partitioning strategies based on such a cost model to achieve an even partition with explainable quality. We implement our framework upon Apache Spark platform and conduct extensive experiments on four real-world datasets. The results show that our method significantly outperforms state-of-the-art methods.
Efficient Similarity Search on Quasi-Metric Graphs

Tianming Zhang,Yunjun Gao,Lu Chen,Guanlin Chen,Shiliang Pu

DOI: https://doi.org/10.1109/access.2019.2930753

IF: 3.9

2019-01-01

IEEE Access

Abstract:Similarity search in metric spaces finds similar objects to a given object, which has received much attention as it is able to support various data types and flexible similarity metrics. In real-life applications, metric spaces might be combined with graphs, resulting in geo-social network, citation graph, social image graph, to name but a few. In this paper, we introduce a new notion called quasi-metric graph that connects metric data using a graph, and formulate similarity search on quasi-metric graphs based on the combined similarity metric considering both the metric data similarity and graph similarity. We propose two simple efficient approaches, the best-first method and the breadth-first method, which traverse the quasi-metric graph following the best-first and the breadth-first paradigms, respectively, and utilize the triangle inequality to prune unnecessary evaluation. Extensive experiments with three real datasets demonstrate, compared with several baseline methods, the effectiveness and efficiency of our proposed methods.
Top-k similarity join over multi-valued objects

Wenjie Zhang,Jing Xu,Xin Liang,Ying Zhang,Xuemin Lin

DOI: https://doi.org/10.1007/978-3-642-29038-1_37

2012-01-01

Abstract:The top-k similarity joins have been extensively studied and used in a wide spectrum of applications such as information retrieval, decision making, spatial data analysis and data mining. Given two sets of objects U and V, a top-k similarity join returns k pairs of most similar objects from U × V. In the conventional model of top-k similarity join processing, an object is usually regarded as a point in a multi-dimensional space and the similarity between two objects is usually measured by distance metrics such as Euclidean distance. However, in many applications an object may be described by multiple values (instances) and the conventional model is not applicable since it does not address the distributions of object instances. In this paper, we study top-k similarity join queries over multi-valued objects. We apply quantile based distance to explore the relative instance distribution among the multiple instances of objects. Efficient and effective techniques to process top-k similarity joins over multi-valued objects are developed following a filtering-refinement framework. Novel distance, statistic and weight based pruning techniques are proposed. Comprehensive experiments on both real and synthetic datasets demonstrate the efficiency and effectiveness of our techniques.
MELODY-JOIN: Efficient Earth Mover's Distance Similarity Joins Using MapReduce

Jin Huang,Rui Zhang,Rajkumar Buyya,Jian Chen

DOI: https://doi.org/10.1109/icde.2014.6816702

2014-01-01

Abstract:The Earth Mover's Distance (EMD) similarity join retrieves pairs of records with EMD below a given threshold. It has a number of important applications such as near duplicate image retrieval and pattern analysis in probabilistic datasets. However, the computational cost of EMD is super cubic to the number of bins in the histograms used to represent the data objects. Consequently, the EMD similarity join operation is prohibitive for large datasets. This is the first paper that specifically addresses the EMD similarity join and we propose to use MapReduce to approach this problem. The MapReduce algorithms designed for generic metric distance similarity joins are inefficient for the EMD similarity join because they involve a large number of distance computations and have unbalanced workloads on reducers when dealing with skewed datasets. We propose a novel framework, named MELODY-JOIN, which transforms data into the space of EMD lower bounds and performs pruning and partitioning at a low cost because computing these EMD lower bounds has a constant complexity. Furthermore, we address two key problems, the limited pruning power and the unbalanced workloads, by enhancing each phase in the MELODY-JOIN framework. We conduct extensive experiments on real datasets. The results show that MELODY-JOIN outperforms the state-of-the-art technique by an order of magnitude, scales up better on large datasets than the state-of-the-art technique, and scales out well on distributed machines.
Heads-Join: Efficient Earth Mover's Distance Similarity Joins on Hadoop

Jin Huang,Rui Zhang,Rajkumar Buyya,Jian Chen,Yongwei Wu

DOI: https://doi.org/10.1109/tpds.2015.2462354

IF: 5.3

2016-01-01

IEEE Transactions on Parallel and Distributed Systems

Abstract:The Earth Mover's Distance (EMD) similarity join has a number of important applications such as near duplicate image retrieval and distributed based pattern analysis. However, the computational cost of EMD is super cubic and consequently the EMD similarity join operation is prohibitive for datasets of even medium size. We propose to employ the Hadoop platform to speed up the operation. Simply porting the state-of-the-art metric distance similarity join algorithms to Hadoop results in inefficiency because they involve excessive distance computations and are vulnerable to skewed data distributions. We propose a novel framework, named Heads-Join, which transforms data into the space of EMD lower bounds and performs pruning and partitioning at a low cost because computing these EMD lower bounds has constant or linear complexity. We investigate both range and top-$k$ joins, and design efficient algorithms on three popular Hadoop computation paradigms, i.e., MapReduce, Bulk Synchronous Parallel, and Spark. We conduct extensive experiments on both real and synthetic datasets. The results show that Heads-Join outperforms the state-of-the-art metric similarity join technique, i.e., Quickjoin, by up to an order of magnitude and scales out well.
Massjoin: A Mapreduce-Based Method for Scalable String Similarity Joins

Dong Deng,Guoliang Li,Shuang Hao,Jiannan Wang,Jianhua Feng

DOI: https://doi.org/10.1109/icde.2014.6816663

2014-01-01

Abstract:String similarity join is an essential operation in data integration. The era of big data calls for scalable algorithms to support large-scale string similarity joins. In this paper, we study scalable string similarity joins using MapReduce. We propose a MapReduce-based framework, called MASSJOIN, which supports both set-based similarity functions and character-based similarity functions. We extend the existing partition-based signature scheme to support set-based similarity functions. We utilize the signatures to generate key-value pairs. To reduce the transmission cost, we merge key-value pairs to significantly reduce the number of key-value pairs, from cubic to linear complexity, while not sacrificing the pruning power. To improve the performance, we incorporate "light-weight" filter units into the key-value pairs which can be utilized to prune large number of dissimilar pairs without significantly increasing the transmission cost. Experimental results on real-world datasets show that our method significantly outperformed state-of-the-art approaches.
An Efficient Partition Based Method for Exact Set Similarity Joins

Dong Deng,Guoliang Li,He Wen,Jianhua Feng

DOI: https://doi.org/10.14778/2856318.2856330

IF: 2.5

2015-01-01

Proceedings of the VLDB Endowment

Abstract:We study the exact set similarity join problem, which, given two collections of sets, finds out all the similar set pairs from the collections. Existing methods generally utilize the prefix filter based framework. They generate a prefix for each set and prune all the pairs whose prefixes are disjoint. However the pruning power is limited, because if two dissimilar sets share a common element in their prefixes, they cannot be pruned. To address this problem, we propose a partition-based framework. We design a partition scheme to partition the sets into several subsets and guarantee that two sets are similar only if they share a common subset. To improve the pruning power, we propose a mixture of the subsets and their 1-deletion neighborhoods (the subset of a set by eliminating one element). As there are multiple allocation strategies to generate the mixture, we evaluate different allocations and design a dynamic-programming algorithm to select the optimal one. However the time complexity of generating the optimal one is O(s3) for a set with size s. To speed up the allocation selection, we develop a greedy algorithm with an approximation ratio of 2. To further reduce the complexity, we design an adaptive grouping mechanism, and the two techniques can reduce the complexity to O(s log s). Experimental results on three real-world datasets show our method achieves high performance and outperforms state-of-the-art methods by 2-5 times.
What Makes Objects Similar: A Unified Multi-Metric Learning Approach.

Han-Jia Ye,De-Chuan Zhan,Xue-Min Si,Yuan Jiang,Zhi-Hua Zhou

DOI: https://doi.org/10.1109/tpami.2018.2829192

IF: 23.6

2018-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Linkages are essentially determined by similarity measures that may be derived from multiple perspectives. For example, spatial linkages are usually generated based on localities of heterogeneous data. Semantic linkages, however, can come from even more properties, such as different physical meanings behind social relations. Many existing metric learning models focus on spatial linkages but leave the rich semantic factors unconsidered. We propose a Unified Multi-Metric Learning (Um$^2$2l) framework to exploit multiple types of metrics with respect to overdetermined similarities between linkages. In Um$^2$2l, types of combination operators are introduced for distance characterization from multiple perspectives, and thus can introduce flexibilities for representing and utilizing both spatial and semantic linkages. Besides, we propose a uniform solver for Um$^2$2l, and the theoretical analysis reflects the generalization ability of Um$^2$2l as well. Extensive experiments on diverse applications exhibit the superior classification performance and comprehensibility of Um$^2$2l. Visualization results also validate its ability to physical meanings discovery.
MSQL: Efficient Similarity Search in Metric Spaces Using SQL.

Wei Lu,Jiajia Hou,Ying Yan,Meihui Zhang,Xiaoyong Du,Thomas Moscibroda

DOI: https://doi.org/10.1007/s00778-017-0481-6

2017-01-01

Abstract:Similarity search is a primitive operation that arises in a large variety of database applications. Typical examples include identifying articles with similar titles, finding similar images and music in a large digital object repository, etc. While there exist a wide spectrum of access methods for similarity queries in metric spaces, a practical solution that can be fully supported by existing RDBMS with high efficiency still remains an open problem. In this paper, we present MSQL, a practical solution for answering similarity queries in metric spaces fully using SQL. To the best of our knowledge, MSQL enables users to find similar objects by submitting SELECT-FROM-WHERE statements only. MSQL provides a uniform indexing scheme based on a standard built-in $B^+$-tree index, with the ability to accelerate the query processing using index seek. Various query optimization techniques are incorporated in MSQL to significantly reduce CPU and I/O cost. We deploy MSQL on top of PostgreSQL. Extensive experiments on various real data sets demonstrate MSQL’s benefits, performing up to two orders of magnitude faster than existing domain-specific SQL-based solutions and being comparable to native solutions.
A Metric Space Based Software Clone Detection Approach

Zhuo Li,Jianling Sun

DOI: https://doi.org/10.1109/icime.2010.5478099

2011-01-01

International Journal of Advancements in Computing Technology

Abstract:Metric space is a set with definition of distance between elements within this set. This paper introduces metric space into code clone detection, and uses the distance within a metric space to measure the similarity level of code. It proposed a process of building up a metric space to detect clones in software system. Based on metric space which is derived from software metric, the clone detection can get more convenience and flexibility. We also exercise the approach in a real industry project.
Unified Spatial Intersection Algorithms Based on Conformal Geometric Algebra

Feng Zhang,Xiaomin Jiang,Xiaoyi Zhang,Yingzhi Wang,Zhenhong Du,Renyi Liu

DOI: https://doi.org/10.1155/2016/7412373

IF: 1.43

2016-01-01

Mathematical Problems in Engineering

Abstract:Conformal Geometric Algebra has been introduced into geographic information science as a mathematical theory because of its advantages in terms of uniform multidimensional representation and computation. The traditional intersection computation between two geometric objects of different types is not unified. In this study, we propose algorithms based on Conformal Geometric Algebra to determine the spatial relationships between geographic objects in a unified manner. The unified representation and intersection computation can be realized for geometric objects of different dimensions. Different basic judgment rules are provided for different simple geometries. The algorithms are designed and implemented using MapReduce to improve the efficiency of the algorithms. From the results of several experiments we provide, the correctness and effectiveness of the algorithms can be verified.
SiMPSON: Efficient Similarity Search in Metric Spaces over P2P Structured Overlay Networks

Quang Hieu Vu,Mihai Lupu,Sai Wu

DOI: https://doi.org/10.1007/978-3-642-03869-3_48

2009-01-01

Abstract:Similarity search in metric spaces over centralized systems has been significantly studied in the database research community. However, not so much work has been done in the context of P2P networks. This paper introduces SiMPSON: a P2P system supporting similarity search in metric spaces. The aim is to answer queries faster and using less resources than existing systems. For this, each peer first clusters its own data using any off-the-shelf clustering algorithms. Then, the resulting clusters are mapped to one-dimensional values. Finally, these one-dimensional values are indexed into a structured P2P overlay. Our method slightly increases the indexing overhead, but allows us to greatly reduce the number of peers and messages involved in query processing: we trade a small amount of overhead in the data publishing process for a substantial reduction of costs in the querying phase. Based on this architecture, we propose algorithms for processing range and kNN queries. Extensive experimental results validate the claims of efficiency and effectiveness of SiMPSON.
Fairest Neighbors: Tradeoffs Between Metric Queries

Magnus Lie Hetland,Halvard Hummel

DOI: https://doi.org/10.48550/arXiv.2108.03621

2021-08-08

Abstract:Metric search commonly involves finding objects similar to a given sample object. We explore a generalization, where the desired result is a fair tradeoff between multiple query objects. This builds on previous results on complex queries, such as linear combinations. We instead use measures of inequality, like ordered weighted averages, and query existing index structures to find objects that minimize these. We compare our method empirically to linear scan and a post hoc combination of individual queries, and demonstrate a considerable speedup.

Data Structures and Algorithms
K-Join: Knowledge-Aware Similarity Join

Zeyuan Shang,Yaxiao Liu,Guoliang Li,Jianhua Feng

DOI: https://doi.org/10.1109/tkde.2016.2601325

2017-01-01

Abstract:Similarity join is a fundamental operation in data cleaning and integration. Existing similarity-join methods utilize the string similarity to quantify the relevance but neglect the knowledge behind the data, which plays an important role in understanding the data. Thanks to public knowledge bases, e.g., Freebase and Yago, we have an opportunity to use the knowledge to improve similarity join. To address this problem, we study knowledge-aware similarity join, which, given a knowledge hierarchy and two collections of objects (e.g., documents), finds all knowledge-aware similar object pairs. To the best of our knowledge, this is the first study on knowledge-aware similarity join. There are two main challenges. The first is how to quantify the knowledge-aware similarity. The second is how to efficiently identify the similar pairs. To address these challenges, we first propose a new similarity metric to quantify the knowledge-aware similarity using the knowledge hierarchy. We then devise a filter-and-verification framework to efficiently identify the similar pairs. We propose effective signature-based filtering techniques to prune large numbers of dissimilar pairs and develop efficient verification algorithms to verify the candidates that are not pruned in the filter step. Experimental results on real-world datasets show that our method significantly outperforms baseline algorithms in terms of both efficiency and effectiveness.

Metric Similarity Joins Using MapReduce

Metric Similarity Joins Using Mapreduce (Extended Abstract)

Metric Similarity Joins Using MapReduce

An Efficient MapReduce Algorithm for Similarity Join in Metric Spaces

Scalable Metric Similarity Join Using MapReduce

Improve Semantic Web Services Discovery Through Similarity Search in Metric Space

Efficient Top-K Similarity Join Processing over Multi-Valued Objects

Improving Distributed Similarity Join in Metric Space with Error-bounded Sampling

Efficient Similarity Search on Quasi-Metric Graphs

Top-k similarity join over multi-valued objects

MELODY-JOIN: Efficient Earth Mover's Distance Similarity Joins Using MapReduce

Heads-Join: Efficient Earth Mover's Distance Similarity Joins on Hadoop

Massjoin: A Mapreduce-Based Method for Scalable String Similarity Joins

An Efficient Partition Based Method for Exact Set Similarity Joins

What Makes Objects Similar: A Unified Multi-Metric Learning Approach.

MSQL: Efficient Similarity Search in Metric Spaces Using SQL.

A Metric Space Based Software Clone Detection Approach

Unified Spatial Intersection Algorithms Based on Conformal Geometric Algebra

SiMPSON: Efficient Similarity Search in Metric Spaces over P2P Structured Overlay Networks

Fairest Neighbors: Tradeoffs Between Metric Queries

K-Join: Knowledge-Aware Similarity Join