Abstract:Memory disaggregation is a promising architecture for modern datacenters that separates compute and memory resources into independent pools connected by ultra-fast networks, which can improve memory utilization, reduce cost, and enable elastic scaling of compute and memory resources. However, existing memory disaggregation solutions based on remote direct memory access (RDMA) suffer from high latency and additional overheads including page faults and code refactoring. Emerging cache-coherent interconnects such as CXL offer opportunities to reconstruct high-performance memory disaggregation. However, existing CXL-based approaches have physical distance limitation and cannot be deployed across racks. In this article, we propose Rcmp, a novel low-latency and highly scalable memory disaggregation system based on RDMA and CXL. The significant feature is that Rcmp improves the performance of RDMA-based systems via CXL, and leverages RDMA to overcome CXL’s distance limitation. To address the challenges of the mismatch between RDMA and CXL in terms of granularity, communication, and performance, Rcmp (1) provides a global page-based memory space management and enables fine-grained data access, (2) designs an efficient communication mechanism to avoid communication blocking issues, (3) proposes a hot-page identification and swapping strategy to reduce RDMA communications, and (4) designs an RDMA-optimized RPC framework to accelerate RDMA transfers. We implement a prototype of Rcmp and evaluate its performance by using micro-benchmarks and running a key-value store with YCSB benchmarks. The results show that Rcmp can achieve 5.2× lower latency and 3.8× higher throughput than RDMA-based systems. We also demonstrate that Rcmp can scale well with the increasing number of nodes without compromising performance.

Thinking More about RDMA Memory Semantics

A Survey of Storage Systems in the RDMA Era

A Physical-Aware Framework for Memory Network Design Space Exploration

Efficiently Enlarging RDMA-Attached Memory with SSD

TeRM: Extending RDMA-Attached Memory with SSD.

RF-RPC: Remote Fetching RPC Paradigm for RDMA-Enabled Network

Toward Effective and Fair RDMA Resource Sharing.

Maximizing the Benefit of RDMA at End Hosts

Pyramid: Revisiting Memory Extension with Remote Accessible Non-Volatile Main Memory.

RFP: When RPC is Faster than Server-Bypass with RDMA.

The Impact of RDMA on Agreement

POSTER: CAVER: Enhancing RDMA Load Balancing by Hunting Less-Congested Paths

MC-RDMA: Improving Replication Performance of RDMA-based Distributed Systems with Reliable Multicast Support

RDMA is Turing complete, we just did not know it yet!

Direct Distributed Memory Access for CMPs

Correct, Fast Remote Persistence

NP-RDMA: Using Commodity RDMA without Pinning Memory

Rcmp: Reconstructing RDMA-Based Memory Disaggregation via CXL

RB2: Narrow the Gap Between RDMA Abstraction and Performance Via a Middle Layer

Using RDMA for Lock Management

RDMAvisor: Toward Deploying Scalable and Simple RDMA as a Service in Datacenters