Abstract:In order to process complex and large-scale graph data numerous distributed graph-parallel computing platforms have been proposed. However, excessive communications among computing nodes in these systems not only aggravate the network I/O workload of the underlying computing hardware systems but may also cause a decrease in runtime performance and scalability. In this paper, we propose and implement a system called Ligraph, which computes large-scale graph data in distributed mode with lightweight communication overhead. Ligraph is similar to PowerGraph system with three new features: (1) a Gather partial sum difference based computing model; (2) a corresponding lightweight Gather communication mechanism; (3) for PageRank-like algorithms Ligraph additionally employs a lightweight synchronizing communication mechanism and an edge direction-aware graph partition strategy proposed by our former work LightGraph, which is specially designed for PageRank-like algorithms. We have conducted extensive experiments using real-world data sets, and our results verified the effectiveness of Ligraph on reducing the communication overhead and improving the runtime performance and the scalability compared with PowerGraph and LightGraph. For example, compared with PowerGraph under Random partition scenario Ligraph can not only reduce up to 35.2 percent of the communication overhead but also cut up to 21.8 percent of the runtime for PageRank algorithm while processing Twitter data set. Our experiment results also demonstrate that compared with several other representative existing systems Ligraph also outperforms them in graph computing rate.

Large-scale graph processing systems: a survey

Large scale graph processing systems: survey and an experimental evaluation

Optimization Factor Analysis Of Large-Scale Join Queries On Different Platforms

A Survey of Distributed Graph Algorithms on Massive Graphs

Big Graph Management Based on Scalable Computing Platforms

The Ubiquity of Large Graphs and Surprising Challenges of Graph Processing: Extended Survey

Evaluation and Analysis of Distributed Graph-Parallel Processing Frameworks

A Distributed Graph-Parallel Computing System with Lightweight Communication Overhead

Survey of External Memory Large-Scale Graph Processing on a Multi-Core System

Graph Computing Systems for Large Scale Graph Analysis

Reviewing Large Graph Computing from a System Perspective

High-Level Programming Abstractions for Distributed Graph Processing

Managing and Mining Large Graphs

A Survey of Graph Pre-processing Methods: From Algorithmic to Hardware Perspectives

Evaluating Large Graph Processing in MapReduce Based on Message Passing

Experimental Analysis of Distributed Graph Systems

Evaluation and Trade-offs of Graph Processing for Cloud Services

Demystifying Graph Databases: Analysis and Taxonomy of Data Organization, System Designs, and Graph Queries

A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications