Abstract:In this paper, we focus on set similarity join on massive probabilistic data using MapReduce, there is no effective approach that can process this problem efficiently. MapReduce is a popular paradigm that can process large volume data more efficiently, in this paper, we proposed two approaches using MapReduce to deal with this task: Hadoop Join by Map Side Pruning and Hadoop Join by Reduce Side Pruning. Hadoop Join by Map Side Pruning uses the sum of the existence probability to filter out the probabilistic sets directly at the Map task side which have no any chance to be similar with any other probabilistic set. Hadoop Join by Reduce Side Pruning uses probability sum based pruning principle and probability upper bound based pruning principle to reduce the candidate pairs at Reduce task side, it can save the comparison cost. Based on the above approaches, we proposed a hybrid solution that employs both Map-side and Reduce-side pruning methods. Finally we implemented the above approaches on Hadoop-0.20.2 and performed comprehensive experiments to their performance, we also test the speedup ratio compared with the naive method: Block Nested Loop Join. The experiment results show that our approaches have much better performance than that of Block Nested Loop Join and also have good scalability. To the best of our knowledge, this is the first work to try to deal with set similarity join on massive probabilistic data problem using MapReduce paradigm, and the approaches proposed in this paper provide a new way to process the massive probabilistic data.

Learning Markov Blanket Bayesian Network for Big Data in MapReduce.

A Method for Hybrid Bayesian Network Structure Learning from Massive Data Using MapReduce

Hybrid Parrallel Bayesian Network Structure Learning from Massive Data Using MapReduce

Massively parallel learning of Bayesian networks with MapReduce for factor relationship analysis

A Method to Enhance the Efficiency of Markov Blanket for BN in Medical Diagnosis

A Parallel and Incremental Approach for Data-Intensive Learning of Bayesian Networks

Data-Intensive Learning Of Uncertain Knowledge

Data-Intensive Inferences Of Large-Scale Bayesian Networks

Three-Fast-Inter Incremental Association Markov Blanket Learning Algorithm

Missing Data Filling Algorithm for Big Data-Based Map-Reduce Technology

Large-scale Data Mining Method based on Clustering Algorithm Combined with MAPREDUCE

RESEARCH ON ROUGH SET AND BAYES-BASED METEOROLOGICAL DATA MINING ON HADOOP PLATFORM

Mining Markov Blankets Without Causal Sufficiency

Set similarity join on massive probabilistic data using MapReduce

A Novel BN Learning Algorithm Based on Block Learning Strategy

MR-ELM: a MapReduce-based framework for large-scale ELM training in big data era

Parallel Graph Pattern Matching in Massive Networks Based on MapReduce

Analysis and Implementation of MapReduce Parallelization of Na?ve Bayes Algorithm

Big Learning with Bayesian Methods

MapReduce for Bayesian Network Parameter Learning using the EM Algorithm

Scaling Bayesian Network Parameter Learning with Expectation Maximization using MapReduce