Abstract:In order to process complex and large-scale graph data numerous distributed graph-parallel computing platforms have been proposed. However, excessive communications among computing nodes in these systems not only aggravate the network I/O workload of the underlying computing hardware systems but may also cause a decrease in runtime performance and scalability. In this paper, we propose and implement a system called Ligraph, which computes large-scale graph data in distributed mode with lightweight communication overhead. Ligraph is similar to PowerGraph system with three new features: (1) a Gather partial sum difference based computing model; (2) a corresponding lightweight Gather communication mechanism; (3) for PageRank-like algorithms Ligraph additionally employs a lightweight synchronizing communication mechanism and an edge direction-aware graph partition strategy proposed by our former work LightGraph, which is specially designed for PageRank-like algorithms. We have conducted extensive experiments using real-world data sets, and our results verified the effectiveness of Ligraph on reducing the communication overhead and improving the runtime performance and the scalability compared with PowerGraph and LightGraph. For example, compared with PowerGraph under Random partition scenario Ligraph can not only reduce up to 35.2 percent of the communication overhead but also cut up to 21.8 percent of the runtime for PageRank algorithm while processing Twitter data set. Our experiment results also demonstrate that compared with several other representative existing systems Ligraph also outperforms them in graph computing rate.

Big Graph Management Based on Scalable Computing Platforms

Optimization Factor Analysis Of Large-Scale Join Queries On Different Platforms

Large-scale graph processing systems: a survey

Managing and Mining Large Graphs

Large scale graph processing systems: survey and an experimental evaluation

A Distributed Graph-Parallel Computing System with Lightweight Communication Overhead

Evaluating Large Graph Processing in MapReduce Based on Message Passing

A Survey of Distributed Graph Algorithms on Massive Graphs

Efficient Subgraph Matching on Billion Node Graphs

Graph Computing Systems for Large Scale Graph Analysis

Evaluation and Analysis of Distributed Graph-Parallel Processing Frameworks

A Parallel Computing Model for Large-Graph Mining with MapReduce.

How to Partition a Billion-Node Graph

Superblock: An Application-Aware Dynamic Partition Strategy for Large-Scale Graph

Graph Processing Framework Supporting Elastic Scalability in Distributed Shared Environment

Evaluation and Trade-offs of Graph Processing for Cloud Services

Execution Feature Extraction and Prediction for Large-Scale Graph Processing Applications

Survey of External Memory Large-Scale Graph Processing on a Multi-Core System

A Feasible Graph Partition Framework for Parallel Computing of Big Graph