Towards Efficient Graph Processing in Geo-Distributed Data Centers
Feng Yao,Qian Tao,Shengyuan Lin,Yanfeng Zhang,Wenyuan Yu,Shufeng Gong,Qiange Wang,Ge Yu,Jingren Zhou
DOI: https://doi.org/10.1109/tpds.2024.3453872
IF: 5.3
2024-09-27
IEEE Transactions on Parallel and Distributed Systems
Abstract:Iterative graph processing is widely used as a significant paradigm for large-scale data analysis. In many global businesses of multinational enterprises, graph-structure data is usually geographically distributed in different regions to support low-latency services. Geo-distributed graph processing suffers from the Wide Area Networks (WANs) with scarce and heterogeneous bandwidth, thus essentially differs from traditional distributed graph processing. In this paper, we propose RAGraph, a Region-Aware framework for geo-distributed graph processing. At the core of RAGraph, we design a region-aware graph processing framework that allows advancing inefficient global updates locally and enables sensible coordination-free message interactions and flexible replaceable communication module. In terms of graph data preprocessing, RAGraph introduces a contribution-driven edge migration algorithm to effectively utilize network resources. RAGraph also contains an adaptive hierarchical message interaction engine to switch interaction modes adaptively based on network heterogeneity and fluctuation, and a discrepancy-aware message filtering strategy to filter important messages. Experimental results show that RAGraph can achieve an average speedup of 9.7× (up to 98×) and an average WAN cost reduction of 78.5 % (up to 97.3 %) compared with state-of-the-art systems.
computer science, theory & methods,engineering, electrical & electronic