Abstract:With the huge demand for graph analytics in many real-world applications, massive iterative graph processing jobs are concurrently performed on the same graphs and suffer from significant high data access cost. To lower the data access cost toward high performance, several out-of-core concurrent graph processing solutions are recently designed to handle concurrent jobs by enabling these jobs to share the accesses of the same graph data. However, the set of active vertices in each partition are usually different for various concurrent jobs and also evolve with time, where some high-degree ones (or called hub-vertices) of these active vertices require more iterations to converge due to the power-law property of real-world graphs. In consequence, existing solutions still suffer from much unnecessary I/O traffic, because they have to entirely load each partition into the memory for concurrent jobs even if most vertices in this partition are inactive and may be shared by a few jobs. In this paper, we propose an efficient structure-aware storage system, called GraphSO, for higher throughput of the execution of concurrent graph processing jobs. It can be integrated into existing out-of-core graph processing systems to promote the execution efficiency of concurrent jobs with lower I/O overhead. The key design of GraphSO is a fine-grained storage management scheme. Specifically, it logically divides the partitions of existing graph processing systems into a series of small same-sized chunks. At runtime, these small chunks with active vertices are judiciously loaded by GraphSO to construct new logical partitions (i.e., each logical partition is a subset of active chunks) for existing graph processing systems to handle, where the most-frequently-used chunks are preferentially loaded to construct the logical partitions and the other ones are delayed to wait to be required by more jobs. In this way, it can effectively spare the cost of loading the graph data as-ociated with the inactive vertices with low repartitioning overhead and can also enable the loaded graph data to be fully shared by concurrent jobs. Moreover, GraphSO also designs a buffering strategy to efficiently cache the most-frequently-used chunks in the main memory to further minimize the I/O traffic by avoiding repeated load of them. Experimental results show that GraphSO improves the throughput of GridGraph, GraphChi, X-Stream, DynamicShards, LUMOS, Graphene, and Wonderland by 1.4-3.5 times, 2.1-4.3 times, 1.9-4.1 times, 1.9-2.9 times, 1.5-3.1 times, 1.3-1.5 times, and 1.3-2.7 times after integrating with them, respectively.

Distributed Graph Layout for Scalable Small-world Network Analysis

A Distributed Graph-Parallel Computing System with Lightweight Communication Overhead

LightGraph: Lighten Communication in Distributed Graph-Parallel Processing

Scalable Readability Evaluation for Graph Layouts: 2D Geometric Distributed Algorithms

Distributed Shortest Distance Labeling on Large-Scale Graphs

DRGraph: An Efficient Graph Layout Algorithm for Large-scale Graphs by Dimensionality Reduction

Graph3S: A Simple, Speedy and Scalable Distributed Graph Processing System

Two-Dimensional Balanced Partitioning and Efficient Caching for Distributed Graph Analysis

ScaleG: A Distributed Disk-based System for Vertex-centric Graph Processing

A Closer Look at Lightweight Graph Reordering

Distributed Data Placement via Graph Partitioning

Towards Fast Large-scale Graph Analysis via Two-dimensional Balanced Partitioning.

Scaling Distance Labeling on Small-World Networks

Memory Hierarchy Sensitive Graph Layout

Accelerating network layouts using graph neural networks

3-D Partitioning for Large-Scale Graph Processing.

xDGP: A Dynamic Graph Processing System with Adaptive Partitioning

A Structure-aware Storage Optimization for Out-of-Core Concurrent Graph Processing

Distributed Balanced Partitioning via Linear Embedding

A regularized graph layout framework for dynamic network visualization

DHPV: a distributed algorithm for large-scale graph partitioning