Abstract:Data processing engines increasingly leverage distributed file systems for scalable, cost-effective storage. While the Apache Parquet columnar format has become a popular choice for data storage and retrieval, the immutability of Parquet files renders it impractical to meet the demands of frequent updates in contemporary analytical workloads. Log-Structured Tables (LSTs), such as Delta Lake, Apache Iceberg, and Apache Hudi, offer an alternative for scenarios requiring data mutability, providing a balance between efficient updates and the benefits of columnar storage. They provide features like transactions, time-travel, and schema evolution, enhancing usability and enabling access from multiple engines. Moreover, engines like Apache Spark and Trino can be configured to leverage the optimizations and controls offered by LSTs to meet specific business needs. Conventional benchmarks and tools are inadequate for evaluating the transformative changes in the storage layer resulting from these advancements, as they do not allow us to measure the impact of design and optimization choices in this new setting. In this paper, we propose a novel benchmarking approach and metrics that build upon existing benchmarks, aiming to systematically assess LSTs. We develop a framework, LST-Bench, which facilitates effective exploration and evaluation of the collaborative functioning of LSTs and data processing engines through tailored benchmark packages. A package is a mix of use patterns reflecting a target workload; LST-Bench makes it easy to define a wide range of use patterns and combine them into a package, and we include a baseline package for completeness. Our assessment demonstrates the effectiveness of our framework and benchmark packages in extracting valuable insights across diverse environments. The code for LST-Bench is open-sourced and is available at <a class="link-external link-https" href="https://github.com/microsoft/lst-bench/" rel="external noopener nofollow">this https URL</a> .

The Linked Data Benchmark Council (LDBC): Driving competition and collaboration in the graph data management space

The LDBC Financial Benchmark

The LDBC Social Network Benchmark Interactive workload v2: A transactional graph query benchmark with deep delete operations

Implementing the draft Graph Query Language Standard

LITMUS: An Open Extensible Framework for Benchmarking RDF Data Management Solutions

Benchmarking Graph Data Management and Processing Systems: A Survey

4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBs

Dagstuhl Manifestos, Vol. 7, Issue 1 ISSN 2193-2433

An Empirical Study on Recent Graph Database Systems.

LUBM4OBDA: Benchmarking OBDA Systems with Inference and Meta Knowledge

LST-Bench: Benchmarking Log-Structured Tables in the Cloud

OCDB: Revisiting Causal Discovery with a Comprehensive Benchmark and Evaluation Framework

DiscoveryBench: Towards Data-Driven Discovery with Large Language Models

OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs

Which Category Is Better: Benchmarking Relational and Graph Database Management Systems

Open Graph Benchmark: Datasets for Machine Learning on Graphs

DBtrends: Publishing and Benchmarking RDF Ranking Functions.

Building a High-Performance Graph Storage on Top of Tree-Structured Key-Value Stores

LakeBench: Benchmarks for Data Discovery over Data Lakes

DBL: Efficient Reachability Queries on Dynamic Graphs (Complete Version)

Benchmarking Data Lakes Featuring Structured and Unstructured Data with DLBench