LEFT: LightwEight and FasT Packet Reordering for RDMA

Peihao Huang,Xin Zhang,Zhigang Chen,Can Liu,Guo Chen

DOI: https://doi.org/10.1145/3663408.3663418

2024-01-01

Abstract:RDMA, as a cutting-edge networking technology, has gained extensive adoption in large-scale data centers due to its exceptional characteristics, such as low and stable latency, high throughput and low CPU utilization. However, due to the limited on-chip memory of the RDMA Network Interface Cards (RNIC), commercial RDMA usually only supports single-path transmission and cannot fully utilize the rich parallel paths within the DCN, resulting in insufficient bandwidth utilization. Multipath transmission can improve bandwidth utilization, but the out-of-order (OoO) packets it brings negatively impacts the performance of RNICs. Recent works have attempted to address this issue by using bitmaps to record OoO packets to better support multipath transmission. However, these approaches either consume excessive memory for maintaining bitmaps, leading to poor connection scalability, or introduce high latency in bitmap sharing. Consequently, implementing efficient packet reordering in RDMA remains a challenge. To this end, we propose LEFT, a fast and lightweight RDMA packet reorder. It is fast due to its ability to process packet reordering at speeds exceeding 200Gbps on the RNIC, and it is scalable by introducing enhanced dual-state shared bitmap schemes, thereby consuming minimal memory even under high concurrency. Specifically, LEFT adopts a fast and slow path of packet reordering to reduce latency when dealing with multipath RTT. Simulation evaluation shows that LEFT maintains a throughput of 94%+ even when the multi-path RTT difference is 32 times, which is 180% higher than the throughput achieved by using an ordinary shared bitmap pool. By adding an average of only 7B of extra on-chip states for each connection, LEFT achieves up to 1.54x higher throughput while reducing 99% tail FCT by 29% compared to single path transmission.