Abstract:Compute eXpress Link (CXL) is a pivotal technology for memory disaggregation in future heterogeneous computing systems, enabling on-demand memory expansion and improved resource utilization. Despite its potential, CXL is in its early stages with limited market products, highlighting the need for a reliable system-level simulation tool. This paper introduces CXL-DMSim, an open-source, high-fidelity full-system simulator for CXL disaggregated memory systems, comparable in speed to gem5. CXL-DMSim includes a flexible CXL memory expander model, device driver, and support for CXLio and CXLmem protocols. It supports both app-managed and kernel-managed modes, with the latter featuring a NUMA-compatible mechanism. Rigorous verification against real hardware testbeds with FPGA-based and ASIC-based CXL memory prototypes confirms CXL-DMSim's accuracy, with an average simulation error of 4.1%. Benchmark results using LMbench and STREAM indicate that CXL-FPGA memory has approximately ~2.88x higher latency than local DDR, while CXL-ASIC latency is about ~2.18x. CXL-FPGA achieves 45-69% of local DDR's memory bandwidth, and CXL-ASIC reaches 82-83%. The performance of CXL memory is significantly more sensitive to Rd/Wr patterns than local DDR, with optimal bandwidth at a 74%:26% ratio rather than 50%:50% due to the current CXL+DDR controller design. The study also shows that CXL memory can markedly enhance the performance of memory-intensive applications, with the most improvement seen in Viper (~23x) and in bandwidth-sensitive scenarios like MERCI (16%). CXL-DMSim's observability and expandability are demonstrated through detailed case studies, showcasing its potential for research on future CXL-interconnected hybrid memory pools.

Exploring and Evaluating Real-world CXL: Use Cases and System Adoption

Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices

An Overview of Computing-in-Memory Interfaces

CXL-Interference: Analysis and Characterization in Modern Computer Systems

A Comprehensive Simulation Framework for CXL Disaggregated Memory

Streamlining CXL Adoption for Hyperscale Efficiency

An Introduction to the Compute Express Link (CXL) Interconnect

CXL Memory as Persistent Memory for Disaggregated HPC: A Practical Approach

Modeling and Benchmarking Computing-in-Memory for Design Space Exploration.

A CXL- Powered Database System: Opportunities and Challenges

Dissecting CXL Memory Performance at Scale: Analysis, Modeling, and Optimization

Toward CXL-Native Memory Tiering Via Device-Side Profiling

Improving key-value cache performance with heterogeneous memory tiering: A case study of CXL-based memory expansion

emucxl: an emulation framework for CXL-based disaggregated memory applications

A Physical-Aware Framework for Memory Network Design Space Exploration

Next-Gen Interconnection Systems with Compute Express Link: a Comprehensive Survey

Optimizing System Memory Bandwidth with Micron CXL Memory Expansion Modules on Intel Xeon 6 Processors

A Full-System Simulation Framework for CXL-Based SSD Memory System

A Programming Model for Disaggregated Memory over CXL

Memory Sharing with CXL: Hardware and Software Design Approaches

NeoMem: Hardware/Software Co-Design for CXL-Native Memory Tiering