Abstract:N-hop neighborhoods information is very useful in analytic tasks on large-scale graphs, like finding clique in a social network, recommending friends or advertising links according to one’s interests, predicting links among websites and etc. To get the N-hop neighborhoods information on a large graph, such as a web graph, a twitter social graph, the most straightforward method is to conduct a breadth first search (BFS) on a parallel distributed graph processing framework, such as Pregel and GraphLab. However, due to the massive volume of message transfer, the BFS method results in high communication cost and has low efficiency. In this work, we propose a key/value based method, namely KVB, which perfectly fits into the prevailing parallel graph processing framework and computes N-hop neighborhoods on a large scale graph efficiently. Unlike the BFS method, our method need not transfer large amount of neighborhoods information, thus, significantly reduces the overhead on both the communication and intermediate results in the distributed framework.We formalize the N-hop neighborhoods query processing as an optimization problem based on a set of quantitative cost metrics of parallel graph processing. Moreover, we propose a solution to efficiently load only the relevant neighborhoods for computation. Specially, we prove the optimal partial neighborhoods load problem is NP-hard and carefully design a heuristic strategy. We have implemented our algorithm on a distributed graph framework- Spark GraphX and validated our solution with extensive experiments over a number of real world and synthetic large graphs on a modest indoor cluster. Experiments show that our solution generally gains an order of magnitude speedup comparing to the state-of-art BFS implementation.

Evaluating Large Graph Processing in MapReduce Based on Message Passing

Parallel Graph Pattern Matching in Massive Networks Based on MapReduce

HadoopM: A Message-Enabled Data Processing System on Large Clusters.

A Parallel Computing Model for Large-Graph Mining with MapReduce.

A Multi-source Message Passing Model to Improve the Parallelism Efficiency of Graph Mining on MapReduce

Towards Scalable Subgraph Pattern Matching over Big Graphs on MapReduce.

Combination of in-memory graph computation with mapreduce: a subgraph-centric method of pagerank

Scalable Big Graph Processing in MapReduce

A Distributed Graph-Parallel Computing System with Lightweight Communication Overhead

Single Large-Scale Graph Frequent Subgraph Algorithm Based on Spark

Large-Scale Social Network Analysis Based on MapReduce

Implementation of Parallel PageRank Algoirthm Based on MapReduce

Study on Calculation Method for Internet Topological Parameters Based on MapReduce

Evaluation and Analysis of Distributed Graph-Parallel Processing Frameworks

Efficient And Scalable Mining Of Frequent Subgraphs Using Distributed Graph Processing Systems

Parallelized Similarity Flooding Algorithm for Processing Large Scale Graph Datasets with MapReduce

A Comparison Study of Graph Data Processing Based on MyBSP and MapReduce

The Research and Design of Parallel Recommendation Algorithm Based on Mapreduce

An Efficient Parallel Algorithm of N-hop Neighborhoods on Graphs in Distributed Environment

Research and Implementation of MapReduce Programming Oriented Graphical Modeling System

High efficiency large-scale graph data processing mechanism in environment of Spark