ByteGraph: A High-Performance Distributed Graph Database in ByteDance
Lei Li,Han Yang,Hongzhi Chen,Chen Zheng,Meng,Ying-Xin Hu,Yu Wang,Ting Ye,Ting Wu,Xin Li,James Cheng,Xiaohui Chen,Xudong Wang,Hengtian Ding,Hui-dong Zhu,Xu-Dong Fu,Dongqing Han,Chao Chen,Xiangchen Li,Hongfei Tan,Li,Changji Li,KANGSHENG;KANGCHENG WANG,Zhenjie Zhang,Shuai Zhang,Meng-Ti Liu
DOI: https://doi.org/10.14778/3554821.3554824
IF: 2.5
2022-08-01
Proceedings of the VLDB Endowment
Abstract:Most products at ByteDance, e.g., TikTok, Douyin, and Toutiao, naturally generate massive amounts of graph data. To efficiently store, query and update massive graph data is challenging for the broad range of products at ByteDance with various performance requirements. We categorize graph workloads at ByteDance into three types: online analytical, transaction, and serving processing, where each workload has its own characteristics. Existing graph databases have different performance bottlenecks in handling these workloads and none can efficiently handle the scale of graphs at ByteDance. We developed ByteGraph to process these graph workloads with high throughput, low latency and high scalability. There are several key designs in ByteGraph that make it efficient for processing our workloads, including edge-trees to store adjacency lists for high parallelism and low memory usage, adaptive optimizations on thread pools and indexes, and geographic replications to achieve fault tolerance and availability. ByteGraph has been in production use for several years and its performance has shown to be robust for processing a wide range of graph workloads at ByteDance.
Computer Science