Abstract:Emerging cloud-native distributed databases rely on local NVMe SSDs to provide high-performance and highavailable data services to many cloud applications. However, the database clusters suffer from low utilization of local storage because of the imbalance between CPU and storage capacities within each node. For instance, the OceanBase distributed database cluster, with hundreds of PB local storage capacity, only utilizes around 40% of its local storage. Although disaggregated storage (EBS) can enhance storage utilization by provisioning the CPU and storage independently on demand, they suffer from performance bottlenecks and high costs. In this paper, we propose LightPool, a high-performance and lightweight storage pool architecture large-scale deployed in the OceanBase clusters, enhancing storage resource utilization. The key idea of LightPool is aggregating cluster storage into a storage pool and enabling unified management. In particular, LightPool adopts NVMe-oF to enable high-performance storage resource sharing among cluster nodes and integrate the storage pool with Kubernetes to achieve flexible management and allocation of storage resources. Furthermore, we design the hot-upgrade and hot-migration mechanisms to enhance the availability of LightPool. We have deployed LightPool on over 8500 nodes in production clusters. Statistics show that LightPool can improve storage resource utilization from about 40% to 65%. Experimental results show that the extra latency from LightPool is only about 2.1 mu s compared to local storage. Compared to OpenEBS, LightPool enhances bandwidth up to 190.9% in microbenchmarks and throughput up to 6.9% in real-world applications. LightPool is the best practice to deploy NVMe-oF (NVMe/TCP) in the production environment. We also discuss important lessons and experiences learned from the development of LightPool.

The LCG POOL Project, General Overview and Project Structure

The POOL Data Storage, Cache and Conversion Mechanism

LightPool: A NVMe-oF-based High-performance and Lightweight Storage Pool Architecture for Cloud-Native Distributed Database

Grid Computing Pool And Its Framework

Research on the SM-Pool Framework for Part Information Sharing

Transparent Persistence with Java Data Objects

Computing Pool: A Simplified and Practical Computational Grid Model.

SEAL: Common Core Libraries and Services for LHC Applications

An Enhanced Generic Pool Framework Supporting Multi-State Transferring

MemPool: A Scalable Manycore Architecture with a Low-Latency Shared L1 Memory

Data Access for LIGO on the OSG

GRAPLEr: A Distributed Collaborative Environment for Lake Ecosystem Modeling that Integrates Overlay Networks, High-throughput Computing, and Web Services

Telepathic Datacenters: Fast RPCs using Shared CXL Memory

PAPooling: Graph-based Position Adaptive Aggregation of Local Geometry in Point Clouds

A Quantitative Approach for Adopting Disaggregated Memory in HPC Systems

Towards automation of computing fabrics using tools from the fabric management workpackage of the EU DataGrid project

High-Performance I/O: HDF5 for Lattice QCD

P2P Resource Pool and Its Application to Optimize Wide-Area Application Level Multicasting

Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service

Performance Analysis of Grid Computing Pool

Flexible, Scalable Mesh and Data Management using PETSc DMPlex