Are Current Benchmarks Adequate to Evaluate Distributed Transactional Databases?
Luyi Qu,Qingshuai Wang,Ting Chen,Keqiang Li,Rong Zhang,Xuan Zhou,Quanqing Xu,Zhifeng Yang,Chuanhui Yang,Weining Qian,Aoying Zhou
DOI: https://doi.org/10.1016/j.tbench.2022.100031
2022-01-01
BenchCouncil Transactions on Benchmarks, Standards and Evaluations
Abstract:With the rapid development of distributed transactional databases in recent years, there is an urgent need for fair performance evaluation and comparison. Though there are various open-source benchmarks built for databases, it is lack of a comprehensive study about the applicability for distributed transactional databases. This paper presents a review of the state-of-art benchmarks with respect to distributed transactional databases. We first summarize the representative architectures of distributed transactional databases and then provide an overview about the chock points in distributed transactional databases. Then, we classify the classic transactional benchmarks based on their characteristics and design purposes. Finally, we review these benchmarks from schema and data definition, workload generation, and evaluation and metrics to check whether they are still applicable to distributed transactional databases with respect to the chock points. This paper exposes a potential research direction to motivate future benchmark designs in the area of distributed transactional databases.