Low-overhead General-purpose Near-Data Processing in CXL Memory Expanders

Hyungkyu Ham,Jeongmin Hong,Geonwoo Park,Yunseon Shin,Okkyun Woo,Wonhyuk Yang,Jinhoon Bae,Eunhyeok Park,Hyojin Sung,Euicheol Lim,Gwangsun Kim

2024-09-23

Abstract:Emerging Compute Express Link (CXL) enables cost-efficient memory expansion beyond the local DRAM of processors. While its CXL.mem protocol provides minimal latency overhead through an optimized protocol stack, frequent CXL memory accesses can result in significant slowdowns for memory-bound applications whether they are latency-sensitive or bandwidth-intensive. The near-data processing (NDP) in the CXL controller promises to overcome such limitations of passive CXL memory. However, prior work on NDP in CXL memory proposes application-specific units that are not suitable for practical CXL memory-based systems that should support various applications. On the other hand, existing CPU or GPU cores are not cost-effective for NDP because they are not optimized for memory-bound applications. In addition, the communication between the host processor and CXL controller for NDP offloading should achieve low latency, but existing <a class="link-external link-http" href="http://CXL.io/PCIe-based" rel="external noopener nofollow">this http URL</a> mechanisms incur $\mu$s-scale latency and are not suitable for fine-grained NDP. To achieve high-performance NDP end-to-end, we propose a low-overhead general-purpose NDP architecture for CXL memory referred to as Memory-Mapped NDP (M$^2$NDP), which comprises memory-mapped functions (M$^2$func) and memory-mapped $\mu$threading (M$^2\mu$thread). M$^2$func is a CXL.mem-compatible low-overhead communication mechanism between the host processor and NDP controller in CXL memory. M$^2\mu$thread enables low-cost, general-purpose NDP unit design by introducing lightweight $\mu$threads that support highly concurrent execution of kernels with minimal resource wastage. Combining them, M$^2$NDP achieves significant speedups for various workloads by up to 128x (14.5x overall) and reduces energy by up to 87.9% (80.3% overall) compared to baseline CPU/GPU hosts with passive CXL memory.

Hardware Architecture

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to implement low - overhead general - purpose Near - Data Processing (NDP) in CXL (Compute Express Link) memory expanders. Specifically, the paper proposes solutions to the following challenges:

1. **Performance limitations of passive CXL memory**: Although the CXL protocol can provide low - latency remote memory access, frequent CXL memory accesses can lead to significant performance degradation in memory - intensive applications, whether they are latency - sensitive applications or bandwidth - intensive applications.

2. **Limitations of existing NDP solutions**:
- **Application - specificity**: Existing NDP solutions are usually designed for specific applications and are not suitable for actual CXL memory systems because these systems need to support multiple applications.
- **Cost - effectiveness**: Existing CPU or GPU cores do not have an advantage in performance and cost when used for NDP because they are not optimized for memory - intensive applications.
- **High communication overhead**: The traditional CXL.io/PCIe mechanism generates microsecond - level latencies when performing NDP offloading, which is not suitable for fine - grained NDP.

To solve these problems, the paper proposes a low - overhead general - purpose NDP architecture called Memory - Mapped NDP (M2NDP), which consists of two key components:

- **Memory - Mapped Functions (M2func)**: It enables low - overhead communication between the host and the NDP controller through the CXL.mem protocol.
- **Memory - Mapped µthreading (M2µthread)**: It supports efficient and general - purpose NDP kernel execution by introducing lightweight µthreads.

Through these innovations, M2NDP can achieve significant performance improvements and energy consumption reductions under various workloads.