Compressed Data Direct Computing for Databases
Weitao Wan,Feng Zhang,Chenyang Zhang,Mingde Zhang,Jidong Zhai,Yunpeng Chai,Huanchen Zhang,Wei Lu,Yuxing Chen,Haixiang Li,Anqun Pan,Xiaoyong Du
DOI: https://doi.org/10.1109/tkde.2023.3316274
IF: 9.235
2023-01-01
IEEE Transactions on Knowledge and Data Engineering
Abstract:Directly performing operations on compressed data has been proven to be a big success facing big data problems in modern data management systems. These systems have demonstrated significant compression benefits and performance improvement for data analytics applications. However, current systems only focus on data queries, while a complete big data system must support both data query and data manipulation. To solve this problem, we develop CompressDB, which is a new storage engine that can support data processing for databases without decompression. CompressDB has the following advantages. First, CompressDB utilizes context-free grammar to compress data, and supports both data query and data manipulation. Second, for adaptability, we integrate CompressDB to file systems so that a wide range of databases can directly use CompressDB without any change. Third, we enable operation pushdown to storage so that we can perform data query and manipulation in storage systems without bringing large data to memory for high efficiency. We validate the efficacy of CompressDB supporting various kinds of database systems, including SQLite, MySQL, LevelDB, MongoDB, ClickHouse, and Neo4j. We evaluate our method using seven real-world datasets with various lengths, structures, and content in both single node and cluster environments. Experiments show that CompressDB achieves 40% throughput improvement and 44% latency reduction, along with 1.75 compression ratio on average.
computer science, information systems, artificial intelligence,engineering, electrical & electronic