Abstract:Achieving low remote memory access latency remains the biggest challenge in realizing memory disaggregation over Ethernet inside the datacenter. We present EDM that tries to overcome this challenge using two key ideas. First, while the existing network protocols for remote memory access over Ethernet, such as TCP/IP and RDMA, are implemented on top of Ethernet's MAC layer, EDM takes a rather radical approach of implementing the entire network protocol stack for remote memory access within the Physical layer (PHY) of the Ethernet. This overcomes fundamental latency and bandwidth overheads imposed by the MAC layer, especially for small memory messages. Second, EDM implements a centralized, fast, in-network traffic scheduler for memory traffic within the PHY of the Ethernet switch. Inspired by the classic Parallel Iterative Matching (PIM) algorithm, the scheduler dynamically reserves bandwidth between compute and memory nodes by creating virtual circuits in the switch's PHY, thus eliminating the queuing delay and layer 2 packet processing delay at the switch for memory traffic, with high bandwidth utilization. Our FPGA testbed shows that EDM's network fabric incurs a latency of only $\sim$300 ns for remote memory access in an unloaded network, which is an order of magnitude lower than state-of-the-art Ethernet-based solutions such as RoCEv2 and comparable to the emerging PCIe-based solutions such as CXL. Larger-scale network simulations show that even at high network loads, EDM's latency is within 1.3$\times$ its unloaded latency.

FPGA Implementation of RDMA-Based Data Acquisition System Over 100 GbE

FPGA Implementation of High-Speed Data Acquisition System for High-Resolution Millimeter Wave Radar.

Design and FPGA Verification of a Novel Reliable Real-Time Data Transfer System

A High-Performance FPGA-Based RoCE v2 RDMA Packet Parser and Generator

A study of UDP and TCP FPGA implementation for data acquisition system

Verification of 400 GbE on an FPGA Platform with Optical Modules

Efficient PC-FPGA Communication over Gigabit Ethernet

Performance profiling and design choices of an RDMA implementation using FPGA devices

The Design of a High Speed and Reliable Data Center Based on FPGA

Front-end RDMA over Converged Ethernet, real-time firmware simulation

Efficient transmission of measurement data from FPGA to embedded system via Ethernet link

A High-Performance and Flexible Architecture for Accelerating SDN on the MPSoC Platform

High-speed data transfer with FPGAs and QSFP+ modules

FABLE-DTS: Hardware-Software Co-Design of a Fast and Stable Data Transmission System for FPGAs

A 400Gbit Ethernet core enabling High Data Rate Streaming from FPGAs to Servers and GPUs in Radio Astronomy

A synchronous Gigabit Ethernet protocol stack for high-throughput UDP/IP applications

A Gigabit, DMA-enhanced Open-Source Ethernet Controller for Mixed-Criticality Systems

Design and implementation of a data acquisition system based on FPGA and Ethernet

EDM: An Ultra-Low Latency Ethernet Fabric for Memory Disaggregation

Design of the SPI Interface of 10-Gigabit Ethernet with FPGA

Implementation of High Speed Ethernet Traffic Generator Based on FPGA