Abstract:Popular distributed graph processing frameworks, such as Pregel and GraphLab, are based on the vertex-centric computation model, where users write their customized Compute function for each vertex to process the data iteratively. Vertices are evenly partitioned among the compute nodes, and the granularity of parallelism of the graph algorithm is normally tuned by adjusting the number of compute nodes. Vertex-centric model splits the computation into phases. Inside one specific phase, the computation proceeds as an embarrassingly parallel process, because no communication between compute nodes incurs. By default, current graph engine only handles one iteration of the algorithm in a phase. However, in this paper, we find that we can also tune the granularity of parallelism, by aggregating the computation of multiple iterations into one phase, which has a significant impact on the performance of the graph algorithm. In the ideal case, if all computations are handled in one phase, the whole algorithm turns into an embarrassingly parallel algorithm and the benefit of parallelism is maximized. Based on this observation, we propose two approaches, a function-based approach and a parameter-based approach, to automatically transform a Pregel algorithm into a new one with tunable granularity of parallelism. We study the cost of such transformation and the trade-off between the granularity of parallelism and the performance. We provide a new direction to tune the performance of parallel algorithms. Finally, the approaches are implemented in our graph processing system, N2, and we illustrate their performance using popular graph algorithms.

Automatic Parallelization of Graph Queries with MapReduce

Evaluating Large Graph Processing in MapReduce Based on Message Passing

Efficient Query Evaluation on Distributed Graphs with Hadoop Environment

A Parallel Computing Model for Large-Graph Mining with MapReduce.

Let High-level Graph Queries Be Parallel Efficient: an Approach over Structural Recursion on Pregel.

Processing UnQL Graph Queries with Pregel

Efficiently extracting frequent subgraphs using MapReduce

Query optimization for massively parallel data processing.

GLog: A high level graph analysis system using MapReduce

Scaling Hop-Based Reachability Indexing for Fast Graph Pattern Query Processing

Combination of in-memory graph computation with mapreduce: a subgraph-centric method of pagerank

Tuning the granularity of parallelism for distributed graph processing

Distributed structural clustering on large graph

QUBLE: Blending Visual Subgraph Query Formulation with Query Processing on Large Networks

Parallelization of Ontology Construction and Fusion Based on MapReduce

RDF Subgraph Query Based on Common Subgraph in Distributed Environment

Derivation of Parallel-Efficient Structural Recursive Functions from Declarative Graph Queries.

Towards Systematic Parallel Programming of Graph Problems Via Tree Decomposition and Tree Parallelism

Efficient Multi-way Theta-Join Processing Using MapReduce

Distributed Reachability Queries on Massive Graphs

Distributed aggregation-based attributed graph summarization for summary-based approximate attributed graph queries