Abstract:In order to process complex and large-scale graph data numerous distributed graph-parallel computing platforms have been proposed. However, excessive communications among computing nodes in these systems not only aggravate the network I/O workload of the underlying computing hardware systems but may also cause a decrease in runtime performance and scalability. In this paper, we propose and implement a system called Ligraph, which computes large-scale graph data in distributed mode with lightweight communication overhead. Ligraph is similar to PowerGraph system with three new features: (1) a Gather partial sum difference based computing model; (2) a corresponding lightweight Gather communication mechanism; (3) for PageRank-like algorithms Ligraph additionally employs a lightweight synchronizing communication mechanism and an edge direction-aware graph partition strategy proposed by our former work LightGraph, which is specially designed for PageRank-like algorithms. We have conducted extensive experiments using real-world data sets, and our results verified the effectiveness of Ligraph on reducing the communication overhead and improving the runtime performance and the scalability compared with PowerGraph and LightGraph. For example, compared with PowerGraph under Random partition scenario Ligraph can not only reduce up to 35.2 percent of the communication overhead but also cut up to 21.8 percent of the runtime for PageRank algorithm while processing Twitter data set. Our experiment results also demonstrate that compared with several other representative existing systems Ligraph also outperforms them in graph computing rate.

On computing PageRank via lumping the Google matrix

Lumping Algorithms for Computing Google’s PageRank and Its Derivative, with Attention to Unreferenced Nodes

Weak dangling block reordering and multi-step block compression for efficiently computing and updating PageRank solutions

On computing HITS ExpertRank via lumping the hub matrix

The Extrapolation-Accelerated Multilevel Aggregation Method in PageRank Computation

Enhanced preprocessed multi-step splitting iterations for computing PageRank

Distributed PageRank Computation Based on Iterative Aggregation-Disaggregation Methods.

An Improved Approach to the PageRank Problems

A Power–Arnoldi Algorithm for Computing PageRank

A Note on the Two-Step Matrix Splitting Iteration for Computing PageRank

An Efficient Elimination Strategy for Solving Pagerank Problems

Off-diagonal Low-Rank Preconditioner for Difficult PageRank Problems

The Improved PageRank Matrix and the New Convergence Criterion

An Adaptive Reordered Method for Computing PageRank

Fully distributed PageRank computation with exponential convergence.

Comments on "jordan Canonical Form of the Google Matrix"

Fast Computation of Stationary Joint Probability Distribution of Sparse Markov Chains

Convergence of Distributed Randomized PageRank Algorithms.

Two Accelerated Non-backtracking PageRank Algorithms for Large-scale Networks

Randomized Incremental Algorithms For The Pagerank Computation

A Distributed Graph-Parallel Computing System with Lightweight Communication Overhead