Abstract:Large scale graph processing represents an interesting challenge due to the characteristics of the graph structure. Generally, a distributed graph processing framework is a better choice for large graphs with billions of edges. However, traditional iterative computation models like BSP under-perform due to large communication overheads and slow iterative convergence in a distributed environment. Here, we present DistPathGraph, a distributed graph processing framework based on PathGraph. First, considering the difference between single machine and cluster, we describe a novel cluster based partitioning method that is different from PathGraph. Second, due to dependence of vertices and consistency of data replicas in different partitions, we present a scheme to control the order of vertices in the updating procedure. Finally, we design a message-packing strategy that improves communication congestion and the rate of iterative convergence. Generally, synchronous communication model controls the computation and communication steps by barrier, which induces more CPU idle time and communication congestion. Although asynchronous model can eliminate CPU idle in some sense, it may lead to inconsistency of vertices and require complex control logic. Comparing to two models, our strategy is a better compromise. We evaluate our graph processing framework against GraphLab. The experimental results validate that our partitioning method and communication strategy improve performance, and that our graph processing framework outperforms GraphLab by up to 6.53X.

Developing the Parallelization Methods for Finding the All-Pairs Shortest Paths in Distributed Memory Architecture

Communication Avoiding All-Pairs Shortest Paths Algorithm for Sparse Graphs

Fast All-Pairs Shortest Paths Algorithm in Large Sparse Graph

Practical Parallel Algorithm for All-Pairs Shortest-Path Problem

Parallel Power Flow Algorithm Based on MPI and P-Q Analysis Method

Evaluation and Analysis of Distributed Graph-Parallel Processing Frameworks

A parallel Dijkstra algorithm based on multi-granularity communication

Optimizations to the Parallel Breath First Search on Distributed Memory

GPU-Based Hybrid Algorithm of All-pairs Shortest Paths

Shortest Path Computation over Disk-Resident Large Graphs Based on Extended Bulk Synchronous Parallel Methods.

Asynchronous Parallel Dijkstra's Algorithm on Intel Xeon Phi Processor - How to Accelerate Irregular Memory Access Algorithm.

A Feasible Graph Partition Framework for Parallel Computing of Big Graph

Exploiting Data Dependency to Mitigate Stragglers in Distributed Spatial Simulation.

Enabling Application-Aware Flexible Graph Partition Mechanism for Parallel Graph Processing Systems.

Distributed PathGraph: A Cluster Centric Framework for Distributed Processing Graph

A parallel graph partitioning algorithm to speed up the large-scale distributed graph mining.

A Feasible Graph Partition Framework for Random Walks Implemented by Parallel Computing in Big Graph

Brief Announcement: A Tight Distributed Algorithm for All Pairs Shortest Paths and Applications.

A Parallel Simulation Framework Incorporating Machine Learning-Based Hotspot Detection for Accelerated Power Grid Analysis

Implementation of A Parallel Graph Partition Algorithm to Speed Up Bsp Computing

Scalable distributed Louvain algorithm for community detection in large graphs