Abstract:In the era of big data, log-structured merge-tree (LSM-tree) based key–value stores (KV stores) have become an effective alternative to the traditional relational database owing to their high performance and scalability. KV stores are widely used to manage big unstructured data for data-intensive applications. A compaction operation is triggered to sort data and compact space, but this can degrade write performance, causing write amplification to increase. The compaction process results in extensive data movement and computation. Recently, near-data processing (NDP) models, which place computation close to data sources, aiming to reduce the cost of data movement between the device and the host, have been widely applied to optimize the performance of KV stores. The existing researches on NDP-based KV stores mainly focus on a single NDP-enabled storage device (NDP device for short). In this paper, we present a storage computing architecture with multiple NDP devices named MStore for accelerating compaction performance in LSM-tree based KV stores. MStore can achieve high performance owing to its storage and computing convergence. It employs a multi-column LSM-tree data layout based on key ranges. Compaction tasks on multiple NDP devices are performed in parallel, improving compaction performance. MStore increases the width of the LSM-tree level to reduce the depth, which reduces the execution time and write amplification of compaction. Following the data organization, we design load balancing mechanisms, including storage load balancing by key range adjustment and computation load balancing across compaction tasks. In addition, we develop a batch read interface and a greedy scheduling scheme for user I/Os. MStore aims to fully utilize the computing and I/O resources to boost the compaction performance in KV stores. Compared with single-device NDP systems (such as TStore), MStore with four NDP devices achieves 3.88 × speedup under random-write DB_Bench. When running YCSB-C with a random-write workload, MStore is 4.32 × faster than TStore. The performance of read and scan on MStore is improved by 2.23× and 2.18×, respectively, compared with TStore. These results have strong implications for the implementations of MStore, and their use as replacements for single NDP based KV stores.

KCGS-Store: A Columnar Storage Based on Group Sorting of Key Columns

Transactional Multi-row Access Guarantee in the Key-Value Store

RCFile: A Fast and Space-Efficient Data Placement Structure in MapReduce-based Warehouse Systems

An Empirical Evaluation of Columnar Storage Formats

Grouping Time Series for Efficient Columnar Storage.

Wide Table Layout Optimization Based on Column Ordering and Duplication

<i>SA-LSM</i>: Optimize Data Layout for LSM-tree Based Storage using Survival Analysis

Balancing Garbage Collection vs I/O Amplification using hybrid Key-Value Placement in LSM-based Key-Value Stores

SA-LSM

FPGA-Accelerated Compactions for LSM-based Key-Value Store.

A storage computing architecture with multiple NDP devices for accelerating compaction performance in LSM-tree based KV stores

A Ceph-based storage strategy for big gridded remote sensing data

Mega-KV

SepStore: Data Storage Accelerator for Distributed File Systems by Separating Small Files from Large Files.

A Distributed In-Memory Key-Value Store System on Heterogeneous CPU–GPU Cluster

FlashKV

Atlas: Baidu's key-value storage system for cloud data

An Efficient Design and Implementation of LSM-tree Based Key-Value Store on Open-Channel SSD.

Column-Oriented Storage Techniques for MapReduce

Qd-tree: Learning Data Layouts for Big Data Analytics

Query Optimization and Rebalancing Methods based on CMD.