Abstract:With the huge demand for graph analytics in many real-world applications, massive iterative graph processing jobs are concurrently performed on the same graphs and suffer from significant high data access cost. To lower the data access cost toward high performance, several out-of-core concurrent graph processing solutions are recently designed to handle concurrent jobs by enabling these jobs to share the accesses of the same graph data. However, the set of active vertices in each partition are usually different for various concurrent jobs and also evolve with time, where some high-degree ones (or called hub-vertices) of these active vertices require more iterations to converge due to the power-law property of real-world graphs. In consequence, existing solutions still suffer from much unnecessary I/O traffic, because they have to entirely load each partition into the memory for concurrent jobs even if most vertices in this partition are inactive and may be shared by a few jobs. In this paper, we propose an efficient structure-aware storage system, called GraphSO, for higher throughput of the execution of concurrent graph processing jobs. It can be integrated into existing out-of-core graph processing systems to promote the execution efficiency of concurrent jobs with lower I/O overhead. The key design of GraphSO is a fine-grained storage management scheme. Specifically, it logically divides the partitions of existing graph processing systems into a series of small same-sized chunks. At runtime, these small chunks with active vertices are judiciously loaded by GraphSO to construct new logical partitions (i.e., each logical partition is a subset of active chunks) for existing graph processing systems to handle, where the most-frequently-used chunks are preferentially loaded to construct the logical partitions and the other ones are delayed to wait to be required by more jobs. In this way, it can effectively spare the cost of loading the graph data as-ociated with the inactive vertices with low repartitioning overhead and can also enable the loaded graph data to be fully shared by concurrent jobs. Moreover, GraphSO also designs a buffering strategy to efficiently cache the most-frequently-used chunks in the main memory to further minimize the I/O traffic by avoiding repeated load of them. Experimental results show that GraphSO improves the throughput of GridGraph, GraphChi, X-Stream, DynamicShards, LUMOS, Graphene, and Wonderland by 1.4-3.5 times, 2.1-4.3 times, 1.9-4.1 times, 1.9-2.9 times, 1.5-3.1 times, 1.3-1.5 times, and 1.3-2.7 times after integrating with them, respectively.

VSIM: Distributed Local Structural Vertex Similarity Calculation on Big Graphs

Distributed High-Dimension Matrix Operation Optimization on Spark

Scalable Single-Source SimRank Computation for Large Graphs.

Towards Distributed Node Similarity Search on Graphs

An efficient similarity search framework for SimRank over large dynamic graphs

Evaluation and Analysis of Distributed Graph-Parallel Processing Frameworks

Balance-aware Distributed String Similarity-Based Query Processing System

A Distributed Graph-Parallel Computing System with Lightweight Communication Overhead

Efficient SimRank-based Similarity Join over Large Graphs.

Evaluating Large Graph Processing in MapReduce Based on Message Passing

Tuning the granularity of parallelism for distributed graph processing

An Effective High-Performance Multiway Spatial Join Algorithm with Spark

Distributed structural clustering on large graph

Thinking Like a Vertex: a Survey of Vertex-Centric Frameworks for Distributed Graph Processing

UniWalk: Unidirectional Random Walk Based Scalable SimRank Computation over Large Graph

Efficient Large Scale Distributed Matrix Computation with Spark.

Parallelized Similarity Flooding Algorithm for Processing Large Scale Graph Datasets with MapReduce

Efficient structural node similarity computation on billion-scale graphs

Vertex dissimilarity in community discovery

An Optimal Locality-Aware Task Scheduling Algorithm Based on Bipartite Graph Modelling for Spark Applications

A Structure-aware Storage Optimization for Out-of-Core Concurrent Graph Processing