Krill: A Compiler and Runtime System for Concurrent Graph Processing

Hongzheng Chen,Minghua Shen,Nong Xiao,Yutong Lu

DOI: https://doi.org/10.1145/3458817.3476159

2021-01-01

Abstract:As a large number of emerging graph applications spread across different domains, the need for processing massive concurrent graph jobs (CGJs) is increasing. However, existing graph processing systems designed for a single job cannot efficiently tackle multiple CUs, where they suffer from interfering memory access patterns and inefficient property management. In this paper, we introduce Krill a compiler and runtime system for processing concurrent graph jobs. We propose an SAP model, which decouples graph structure, algorithm, and property. In the compiler, we propose leveraging the property buffer to easily write and manage property data. In the runtime, system, we propose, a novel technique named graph kernel fusion to reduce memory accesses, which fuses all the jobs and processes them as a whole. Experimental results show our system significantly reduces the number of memory accesses for C,Gjs by more than 6x compared with the baseline, and achieves up to 6.76x speedup with 3.54x shorter response latency than GraphM, the state-of-the-art concurrent graph processing system.