Abstract:With the increasing scale and complexity of high-performance computing systems, the rising failure rate poses significant challenges for RDMA networks that aim for high bandwidth and low latency. RDMA networks require hardware-level end-to-end reliable data transmission services to avoid the high cost of software failure recovery. Tianhe HPC interconnection network adopts a NIC-based RDMA reliable connection protocol, RCP. RCP establishes a connection for each message that enters the NIC and releases it after the transmission is complete. However, this introduces an additional round-trip time RTT connection overhead for each message, which severely impacts the performance of networks dominated by short messages in high-performance computing systems. We have found that utilization of receiver-side connection resources has been consistently low because maintaining message-grained connections on the NIC results in rapid release of connections. Therefore, we propose a lightweight RDMA connection protocol based on post-hoc confirmation, PCP. PCP assumes the receiver has connection resources by default and eliminates the need for confirmation from the receiver before sending a message, thus reducing the connection overhead of almost all messages by one RTT. At the same time, PCP also includes mechanisms to address the special case where the receiver lacks connection resources. Evaluation results demonstrate that PCP significantly optimizes short messages and applications dominated by short messages. Moreover, PCP further reduces the usage of receiver-side connection resources. Additionally, PCP does not experience performance degradation even under large-scale heavy loads and severe endpoint congestion.

R-PFC: Enhancing RDMA Network with Restricted and Fine-grained PFC

FG-PFC: A Fine-Grained PFC Mechanism for Lossless RDMA

S-PFC: Enabling Semi-Lossless RDMA Network with Selective Response to PFC.

P-PFC: Reducing Tail Latency with Predictive PFC in Lossless Data Center Networks

Revisiting network support for RDMA

Minimizing Buffer Utilization for Lossless Inter-DC Links

Less is More: Dynamic and Shared Headroom Allocation in PFC-Enabled Datacenter Networks

Implementation of PFC and RCM for RoCEv2 Simulation in OMNeT++

Re-Architecting Buffer Management in Lossless Ethernet

Re-architecting Congestion Management in Lossless Ethernet.

Bifrost: Extending RoCE for Long Distance Inter-DC Links

Switch-Assistant Loss Recovery for RDMA Transport Control

PACC: A Proactive CNP Generation Scheme for Datacenter Networks

LHCC: Low-Latency and Hi-Precision Congestion Control in RDMA Datacenter Networks

Load Balancing With Multi-Level Signals for Lossless Datacenter Networks

A lightweight RDMA Connection Protocol based on Post-hoc Confirmation

RoUD: Scalable RDMA over UD in Lossy Data Center Networks

FC+: Near-optimal Deadlock-free Expander Data Center Networks

Lightning: A Practical Building Block for RDMA Transport Control

Accelerating End-host Congestion Response using P4 Programmable Switches

CACC: A Congestion-Aware Control Mechanism to Reduce INT Overhead and PFC Pause Delay