COER: A Network Interface Offloading Architecture for RDMA and Congestion Control Protocol Codesign

Ke Wu,Dezun Dong,Weixia Xu
DOI: https://doi.org/10.1145/3660525
IF: 1.444
2024-04-22
ACM Transactions on Architecture and Code Optimization
Abstract:RDMA (Remote Direct Memory Access) networks require efficient congestion control to maintain their high throughput and low latency characteristics. However, congestion control protocols deployed at the software layer suffer from slow response times due to the communication overhead between host hardware and software. This limitation has hindered their ability to meet the demands of high-speed networks and applications. Harnessing the capabilities of rapidly advancing Network Interface Card (NIC) can drive progress in congestion control. Some simple congestion control protocols have been offloaded to RDMA NIC to enable faster detection and processing of congestion. However, offloading congestion control to the RDMA NIC faces a significant challenge in integrating the RDMA transport protocol with advanced congestion control protocols that involve complex mechanisms. We have observed that reservation-based proactive congestion control protocols share strong similarities with RDMA transport protocols, allowing them to integrate seamlessly and combine the functionalities of the transport layer and network layer. In this paper, we present COER, an RDMA NIC architecture that leverages the functional components of RDMA to perform reservations and completes the scheduling of congestion control during the scheduling process of the RDMA protocol. COER facilitates the streamlined development of offload strategies for congestion control techniques, specifically proactive congestion control, on RDMA NIC. We use COER to design offloading schemes for eleven congestion control protocols, which we implement and evaluate using a network emulator with a cycle-accurate RDMA NIC model that can load MPI programs. The evaluation results demonstrate that the architecture of COER does not compromise the original characteristics of the congestion control protocols. Compared to a layered protocol stack approach, COER enables the performance of RDMA networks to reach new heights.
computer science, theory & methods, hardware & architecture
What problem does this paper attempt to address?
### What problem does this paper attempt to solve? This paper primarily addresses the following issues: 1. **Improving the congestion control performance of RDMA networks**: - Modern cluster networks (such as high-performance computing systems and data centers) have an increasing demand for high-speed networks, requiring high throughput, low latency, and low CPU overhead. RDMA technology can meet these demands simultaneously, but existing software-layer congestion control protocols (CC protocols) respond slowly and cannot fully utilize the advantages of RDMA. - The paper proposes a new RDMA NIC architecture, COER, which integrates an active congestion control protocol into the RDMA NIC to achieve faster congestion detection and handling. 2. **Integrating RDMA transport protocols with advanced congestion control protocols**: - Software-layer congestion control protocols perform poorly in high-speed network environments, while hardware NICs have higher processing capabilities and can better support congestion control functions. - The paper finds that active congestion control protocols and RDMA transport protocols have similarities and can seamlessly combine the functionalities of both by improving the RDMA connection mechanism. 3. **Addressing existing challenges**: - Existing commercial RDMA NICs are black boxes, making fundamental improvements difficult. The paper proposes a solution, COER, by deeply analyzing the RDMA NIC architecture, enabling better integration of active congestion control protocols into RDMA NICs. - COER leverages the functional components of RDMA connection management to achieve message-level connections, thereby better supporting active congestion control protocols. Through the above methods, COER not only improves the performance of RDMA networks but also provides a flexible framework for designing and implementing various active congestion control protocols.