Abstract:In order to process complex and large-scale graph data numerous distributed graph-parallel computing platforms have been proposed. However, excessive communications among computing nodes in these systems not only aggravate the network I/O workload of the underlying computing hardware systems but may also cause a decrease in runtime performance and scalability. In this paper, we propose and implement a system called Ligraph, which computes large-scale graph data in distributed mode with lightweight communication overhead. Ligraph is similar to PowerGraph system with three new features: (1) a Gather partial sum difference based computing model; (2) a corresponding lightweight Gather communication mechanism; (3) for PageRank-like algorithms Ligraph additionally employs a lightweight synchronizing communication mechanism and an edge direction-aware graph partition strategy proposed by our former work LightGraph, which is specially designed for PageRank-like algorithms. We have conducted extensive experiments using real-world data sets, and our results verified the effectiveness of Ligraph on reducing the communication overhead and improving the runtime performance and the scalability compared with PowerGraph and LightGraph. For example, compared with PowerGraph under Random partition scenario Ligraph can not only reduce up to 35.2 percent of the communication overhead but also cut up to 21.8 percent of the runtime for PageRank algorithm while processing Twitter data set. Our experiment results also demonstrate that compared with several other representative existing systems Ligraph also outperforms them in graph computing rate.

GraphChi-DB: Simple Design for a Scalable Graph Database System -- on Just a PC

G-Tran: A High Performance Distributed Graph Database with a Decentralized Architecture

Building a High-Performance Graph Storage on Top of Tree-Structured Key-Value Stores

The Graph Database Interface: Scaling Online Transactional and Analytical Graph Workloads to Hundreds of Thousands of Cores

System G Distributed Graph Database

GridGraph: Large-Scale Graph Processing on a Single Machine Using 2-Level Hierarchical Partitioning

NXgraph: an Efficient Graph Processing System on a Single Machine

A Distributed Graph-Parallel Computing System with Lightweight Communication Overhead

Pimiento: A Vertex-Centric Graph-Processing Framework on a Single Machine

A disk I/O optimized system for concurrent graph processing jobs

Wind-Bell Index: Towards Ultra-Fast Edge Query for Graph Databases.

Scalable Mining of Large Disk-Based Graph Databases.

ScaleG: A Distributed Disk-based System for Vertex-centric Graph Processing

Computation of K-Core Decomposition on Giraph

G-SQL: Fast Query Processing via Graph Exploration.

Efficient Processing of Very Large Graphs in a Small Cluster

Distributed Graph Layout for Scalable Small-world Network Analysis

ByteGraph: A High-Performance Distributed Graph Database in ByteDance

MBFGraph: An SSD-Based Analytics System for Evolving Graphs

DFOGraph: an I/O- and Communication-Efficient System for Distributed Fully-out-of-core Graph Processing

Efficient Large Graph Processing with Chunk-Based Graph Representation Model.