Abstract:The byte-addressable non-volatile memory (NVM) is a promising technology since it simultaneously provides DRAM-like performance, disk-like capacity, and persistency. The current NVM deployment with byte-addressability is \em symmetric, where NVM devices are directly attached to servers. Due to the higher density, NVM provides much larger capacity and should be shared among servers. Unfortunately, in the symmetric setting, the availability of NVM devices is affected by the specific machine it is attached to. High availability can be achieved by replicating data to NVM on a remote machine. However, it requires full replication of data structure in local memory --- limiting the size of the working set. This paper rethinks NVM deployment and makes a case for the \em asymmetric byte-addressable non-volatile memory architecture, which decouples servers from persistent data storage. In the proposed \em \anvm architecture, NVM devices (i.e., back-end nodes) can be shared by multiple servers (i.e., front-end nodes) and provide recoverable persistent data structures. The asymmetric architecture, which follows the industry trend of \em resource disaggregation, is made possible due to the high-performance network (e.g., RDMA). At the same time, \anvm leads to a number of key problems such as, still relatively long network latency, persistency bottleneck, and simple interface of the back-end NVM nodes. We build \em \anvm framework based on \anvm architecture that implements: 1) high performance persistent data structure update; 2) NVM data management; 3) concurrency control; and 4) crash-consistency and replication. The key idea to remove persistency bottleneck is the use of \em operation log that reduces stall time due to RDMA writes and enables efficient batching and caching in front-end nodes. To evaluate performance, we construct eight widely used data structures and two transaction applications based on \anvm framework. In a 10-node cluster equipped with real NVM devices, results show that \anvm achieves similar or better performance compared to the best possible symmetric architecture while enjoying the benefits of disaggregation. We found the speedup brought by the proposed optimizations is drastic, --- 5$\sim$12× among all benchmarks.

An Efficient Framework for Implementing Persist Data Structures on Remote NVM.

AsymNVM

A Case for Asymmetric Non-Volatile Memory Architecture

Megalloc: Fast Distributed Memory Allocator for NVM-Based Cluster

Storage-Aware Network Stack for NVM-Assisted Key-Value Store

Rio: Fast B+-Tree Based on Remote Accessible Non-Volatile Memory

System Evaluation of the Intel Optane Byte-addressable NVM

MBSA: a Lightweight and Flexible Storage Architecture for Virtual Machines

Demystifying the Performance of HPC Scientific Applications on NVM-based Memory Systems

High-Performance Remote Data Persisting for Key-Value Stores via Persistent Memory Region

High Performance Data Persistence in Non-Volatile Memory for Resilient High Performance Computing

Pyramid: Revisiting Memory Extension with Remote Accessible Non-Volatile Main Memory.

Empirical Study of Transactional Management for Persistent Memory

RC-NVM: Enabling Symmetric Row and Column Memory Accesses for In-memory Databases

TH-DPMS: Design and Implementation of an RDMA-enabled Distributed Persistent Memory Storage System

Correct, Fast Remote Persistence

Maximizing the Benefit of RDMA at End Hosts

Octopus+: an RDMA-Enabled Distributed Persistent Memory File System

RFP: A Remote Fetching Paradigm for RDMA-Accelerated Systems

NVMExplorer: A Framework for Cross-Stack Comparisons of Embedded Non-Volatile Memories

A Survey of Storage Systems in the RDMA Era