Abstract:N-hop neighborhoods information is very useful in analytic tasks on large-scale graphs, like finding clique in a social network, recommending friends or advertising links according to one’s interests, predicting links among websites and etc. To get the N-hop neighborhoods information on a large graph, such as a web graph, a twitter social graph, the most straightforward method is to conduct a breadth first search (BFS) on a parallel distributed graph processing framework, such as Pregel and GraphLab. However, due to the massive volume of message transfer, the BFS method results in high communication cost and has low efficiency. In this work, we propose a key/value based method, namely KVB, which perfectly fits into the prevailing parallel graph processing framework and computes N-hop neighborhoods on a large scale graph efficiently. Unlike the BFS method, our method need not transfer large amount of neighborhoods information, thus, significantly reduces the overhead on both the communication and intermediate results in the distributed framework.We formalize the N-hop neighborhoods query processing as an optimization problem based on a set of quantitative cost metrics of parallel graph processing. Moreover, we propose a solution to efficiently load only the relevant neighborhoods for computation. Specially, we prove the optimal partial neighborhoods load problem is NP-hard and carefully design a heuristic strategy. We have implemented our algorithm on a distributed graph framework- Spark GraphX and validated our solution with extensive experiments over a number of real world and synthetic large graphs on a modest indoor cluster. Experiments show that our solution generally gains an order of magnitude speedup comparing to the state-of-art BFS implementation.

OnFlyP:An Online Distributed Partition Algorithm for Large Scale Graphs Based on Edge-Exchange Model

Local Graph Edge Partitioning

Practical and high-quality partitioning algorithm for large-scale and time-evolving graphs

A Distributed Graph-Parallel Computing System with Lightweight Communication Overhead

Group Reassignment for Dynamic Edge Partitioning

Edge Cluster Based Large Graph Partitioning and Iterative Processing in BSP

Two-Dimensional Balanced Partitioning and Efficient Caching for Distributed Graph Analysis

LightGraph: Lighten Communication in Distributed Graph-Parallel Processing

A Simple Yet Effective Balanced Edge Partition Model for Parallel Computing

VSEP: A Distributed Algorithm for Graph Edge Partitioning

DiterGraph: Toward I/O-Efficient Incremental Computation over Large Graphs with Billion Edges.

A Feasible Graph Partition Framework for Parallel Computing of Big Graph

A parallel graph partitioning algorithm to speed up the large-scale distributed graph mining.

Real-time Edge Repartitioning for Dynamic Graph

WindGP: Efficient Graph Partitioning on Heterogenous Machines

A Two-Phase Method to Balance the Result of Distributed Graph Repartitioning

Local Graph Edge Partitioning with a Two-Stage Heuristic Method.

Grapha: Adaptive Partitioning For Natural Graphs

An Efficient Parallel Algorithm of N-hop Neighborhoods on Graphs in Distributed Environment

Towards Fast Large-scale Graph Analysis via Two-dimensional Balanced Partitioning.

An Efficient Method for Graph Repartitioning in Distributed Environments.