Abstract:LSM-Tree is widely adopted for storing time-series data in Internet of Things. According to conventional policy (denoted by $\pi_{c}$ ), when writing, the data will first be buffered in MemTable in memory. When it is full, the data will be written to the disk to form SSTables. Compaction is triggered to sort the data in each layer of the LSM-Tree on the disk. However, the arrival of data can be unordered due to reasons such as transition delay. Apache IoTDB uses in-order and out-of-order MemTables to separately buffer the in-order and out-of-order data to accelerate queries, namely the separation policy (denoted by $\pi_{s}$ ). However, given a specific space of memory budget to buffer the data, write amplification (WA) of the leveled LSM-Tree will be influenced by $\pi_{s}$ . Whether the influence by separation is positive or negative, and how intense WA is influenced, depend on the properties of workloads and the capacity of the in-order and out-of-order MemTables. It is highly demanded to build robust models for estimating the expected amount of data rewritten in each compaction, and predicting the WA under $\pi_{c}$ and $\pi_{s}$ . Note that as an industrial paper, rather than proposing novel techniques for research problems, we focus on the practice of whether separating or not for lower write amplification. Experiments on synthetic and real-world datasets show that the models for estimating WA are accurate under various delay distributions. In addition, based on the estimation models, we implement an analyzer module in the open-source Apache IoTDB, for choosing the policy with lower WA. We apply the method in the use case of our industrial partner, a service provider of engineering machinery. The use case verifies the effectiveness of deciding whether separation or not by WA estimation.

Distance-Based Outlier Query Optimization in Apache IoTDB

Apache IoTDB: A Time Series Database for IoT Applications

Apache IoTDB: time-series database for internet of things

Backward-Sort for Time Series in Apache IoTDB.

IoTDQ: an Industrial IoT Data Analysis Library for Apache IoTDB

Time-tired compaction: An elastic compaction scheme for LSM-tree based time-series database

Time Series Representation for Visualization in Apache IoTDB

On Reducing Space Amplification with Multi-Column Compaction in Apache IoTDB

Optimizing Time Series Queries with Versions

TsQuality: Measuring Time Series Data Quality in Apache IoTDB

MOST: Model-Based Compression with Outlier Storage for Time Series Data

Time series data encoding in Apache IoTDB: comparative analysis and recommendation

The Design of Apache IoTDB Distributed Framework

Distributed scheduling and storage scheme based on LSM-OCTree for spatiotemporal stream

Separation or Not: on Handing Out-of-Order Time-Series Data in Leveled LSM-Tree

Benchmarking Time Series Databases with IoTDB-Benchmark for IoT Scenarios

Frequency Domain Data Encoding in Apache IoTDB.

Grouping Time Series for Efficient Columnar Storage.

Benchmark Time Series Database with IoTDB-Benchmark for IoT Scenarios.

Mining Distance-Based Outliers from Large Databases in Any Metric Space

A Fast Lightweight Time-Series Store for IoT Data