Abstract:In order to process complex and large-scale graph data numerous distributed graph-parallel computing platforms have been proposed. However, excessive communications among computing nodes in these systems not only aggravate the network I/O workload of the underlying computing hardware systems but may also cause a decrease in runtime performance and scalability. In this paper, we propose and implement a system called Ligraph, which computes large-scale graph data in distributed mode with lightweight communication overhead. Ligraph is similar to PowerGraph system with three new features: (1) a Gather partial sum difference based computing model; (2) a corresponding lightweight Gather communication mechanism; (3) for PageRank-like algorithms Ligraph additionally employs a lightweight synchronizing communication mechanism and an edge direction-aware graph partition strategy proposed by our former work LightGraph, which is specially designed for PageRank-like algorithms. We have conducted extensive experiments using real-world data sets, and our results verified the effectiveness of Ligraph on reducing the communication overhead and improving the runtime performance and the scalability compared with PowerGraph and LightGraph. For example, compared with PowerGraph under Random partition scenario Ligraph can not only reduce up to 35.2 percent of the communication overhead but also cut up to 21.8 percent of the runtime for PageRank algorithm while processing Twitter data set. Our experiment results also demonstrate that compared with several other representative existing systems Ligraph also outperforms them in graph computing rate.

GLog: A high level graph analysis system using MapReduce

GLogS: Interactive Graph Pattern Matching Query At Large Scale.

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

A Distributed Graph-Parallel Computing System with Lightweight Communication Overhead

Evaluating Large Graph Processing in MapReduce Based on Message Passing

Evaluation and Analysis of Distributed Graph-Parallel Processing Frameworks

GRADOOP: Scalable Graph Data Management and Analytics with Hadoop

Loggp: A Log-Based Dynamic Graph Partitioning Method

G-Tran: A High Performance Distributed Graph Database with a Decentralized Architecture

G-SQL: Fast Query Processing via Graph Exploration.

Graph Programming Interface (GPI): A Linear Algebra Programming Model for Large Scale Graph Computations

Design and Implementation of Log Data Analysis Management System Based on Hadoop

VloGraph: A Virtual Knowledge Graph Framework for Distributed Security Log Analysis

Path Pattern Query Processing on Large Graphs

System G Distributed Graph Database

Towards a Converged Relational-Graph Optimization Framework

Towards Efficient Graph Processing in Geo-Distributed Data Centers

Automatic Parallelization of Graph Queries with MapReduce

Experimental Analysis of Distributed Graph Systems

Efficient Query Evaluation on Distributed Graphs with Hadoop Environment

A Modular Graph-Native Query Optimization Framework