Abstract:N-hop neighborhoods information is very useful in analytic tasks on large-scale graphs, like finding clique in a social network, recommending friends or advertising links according to one’s interests, predicting links among websites and etc. To get the N-hop neighborhoods information on a large graph, such as a web graph, a twitter social graph, the most straightforward method is to conduct a breadth first search (BFS) on a parallel distributed graph processing framework, such as Pregel and GraphLab. However, due to the massive volume of message transfer, the BFS method results in high communication cost and has low efficiency. In this work, we propose a key/value based method, namely KVB, which perfectly fits into the prevailing parallel graph processing framework and computes N-hop neighborhoods on a large scale graph efficiently. Unlike the BFS method, our method need not transfer large amount of neighborhoods information, thus, significantly reduces the overhead on both the communication and intermediate results in the distributed framework.We formalize the N-hop neighborhoods query processing as an optimization problem based on a set of quantitative cost metrics of parallel graph processing. Moreover, we propose a solution to efficiently load only the relevant neighborhoods for computation. Specially, we prove the optimal partial neighborhoods load problem is NP-hard and carefully design a heuristic strategy. We have implemented our algorithm on a distributed graph framework- Spark GraphX and validated our solution with extensive experiments over a number of real world and synthetic large graphs on a modest indoor cluster. Experiments show that our solution generally gains an order of magnitude speedup comparing to the state-of-art BFS implementation.

Experimental Analysis of Distributed Graph Systems

Evaluation and Analysis of Distributed Graph-Parallel Processing Frameworks

A Distributed Graph-Parallel Computing System with Lightweight Communication Overhead

Study on Resource Efficiency of Distributed Graph Processing

An Empirical Comparison of Big Graph Frameworks in the Context of Network Analysis

Large scale graph processing systems: survey and an experimental evaluation

Graph3S: A Simple, Speedy and Scalable Distributed Graph Processing System

Graph Computing Systems for Large Scale Graph Analysis

Distributed Graph Processing - Techniques and Systems.

Survey on Technologies of Distributed Graph Processing Systems

An Efficient Parallel Algorithm of N-hop Neighborhoods on Graphs in Distributed Environment

Benchmarking of Distributed Computing Engines Spark and GraphLab for Big Data Analytics

LightGraph: Lighten Communication in Distributed Graph-Parallel Processing

Energy Efficiency of Large Scale Graph Processing Platforms

Large-scale graph processing systems: a survey

GraphA: Efficient Partitioning and Storage for Distributed Graph Computation

High-Level Programming Abstractions for Distributed Graph Processing

A Survey of Distributed Graph Algorithms on Massive Graphs

HPGraph: A High Parallel Graph Processing System Based on Flash Array