Scalable and Reliable RDMA.

Youmin Chen,Jiwu Shu,Youyou Lu

DOI: https://doi.org/10.1145/3267809.3275457

2018-01-01

Abstract:No abstract available.

What problem does this paper attempt to address?

Scalable RDMA RPC on Reliable Connection with Efficient Resource Sharing

Youmin Chen,Youyou Lu,Jiwu Shu

DOI: https://doi.org/10.1145/3302424.3303968

2019-01-01

Abstract:RDMA provides extremely low latency and high bandwidth to distributed systems. Unfortunately, it fails to scale and suffers from performance degradation when transferring data to an increasing number of targets on Reliable Connection (RC). We observe that the above scalability issue has its root in the resource contention in the NIC cache, the CPU cache and the memory of each server. In this paper, we propose ScaleRPC, an efficient RPC primitive using one-sided RDMA verbs on reliable connection to provide scalable performance. To effectively alleviate the resource contention, ScaleRPC introduces 1) connection grouping to organize the network connections into groups, so as to balance the saturation and thrashing of the NIC cache; 2) virtualized mapping to enable a single message pool to be shared by different groups of connections, which reduces CPU cache misses and improve memory utilization. Such scalable connection management provides substantial performance benefits: By deploying ScaleRPC both in a distributed file system and a distributed transactional system, we observe that it achieves high scalability and respectively improves performance by up to 90% and 160% for metadata accessing and SmallBank transaction processing.
Optimizing Distributed Systems with Remote Direct Memory Access

WEI Xingda,CHEN Rong,CHEN Haibo

DOI: https://doi.org/10.11959/j.issn.2096-0271.2018036

IF: 3.3

2018-01-01

Big Data Research

Abstract:Fast network devices with RDMA support have been price-compatible with traditional network primitives such as Ethernet,and it’s now widely deployed in modern data centers.RDMA can be used in two ways.Firstly,it can optimize the messaging primitive in distributed applications.The second way is to redesign the applications with RDMA’s onesided features.One-sided features provide high CPU utilizations and high network performance,but the system should be redesigned.The research progress of RDMA was introduced.An overview on the research efforts on using RDMA for distributed systems was presented.The works on how to use RDMA to redesign systems and the works on how to better leverage RDMA were included.The future research directions were also put forward.
MC-RDMA: Improving Replication Performance of RDMA-based Distributed Systems with Reliable Multicast Support

Chengyuan Huang,Yixiao Gao,Wei Chen,Duoxing Li,Yibo Xiao,Ruyi Zhang,Chen Tian,Xiaoliang Wang,Wanchun Dou,Guihai Chen,Yi Wang,Fu Xiao

DOI: https://doi.org/10.1109/icnp59255.2023.10355619

2023-01-01

Abstract:Remote Direct Memory Access has been widely adopted in distributed storage systems. However, it only supports unicast operations, which degrades the performance significantly for data replication because of bandwidth waste and CPU overhead. To address the problem, we propose MC-RDMA, a distributed and reliable multicast RDMA. It is compatible with existing unicast RDMA but supports lazy packet replication with reliable RDMA multicasting. The key idea of MC-RDMA is utilizing in-network programmable switches to build a NIC-transparent reliable multicast protocol for RDMA. MC-RDMA combines the address information of the IP and RoCEv2 into a sender-initialized multicast routing protocol. Besides, it synchronizes the hardware transmission states of multiple receivers by merging ACKs and NAKs. To verify the effectiveness of MC-RDMA, we implement it with Mellanox ConnectX-6 commodity RNICs and Intel Tofino P4 programmable switches. Experimental results show that MC-RDMA can double the sender bandwidth utilization and reduce the CPU overhead significantly compared to unicast-based RDMA replications. Moreover, it reduces the storage request latency by -30% with realistic workloads and decreases the training time by -50% in the distributed training system.
RDMAvisor: Toward Deploying Scalable and Simple RDMA as a Service in Datacenters

Zhi Wang,Xiaoliang Wang,Zhuzhong Qian,Baoliu Ye,Sanglu Lu

DOI: https://doi.org/10.48550/arXiv.1802.01870

2018-02-06

Abstract:RDMA is increasingly adopted by cloud computing platforms to provide low CPU overhead, low latency, high throughput network services. On the other hand, however, it is still challenging for developers to realize fast deployment of RDMA-aware applications in the datacenter, since the performance is highly related to many lowlevel details of RDMA operations. To address this problem, we present a simple and scalable RDMA as Service (RaaS) to mitigate the impact of RDMA operational details. RaaS provides careful message buffer management to improve CPU/memory utilization and improve the scalability of RDMA operations. These optimized designs lead to simple and flexible programming model for common and knowledgeable users. We have implemented a prototype of RaaS, named RDMAvisor, and evaluated its performance on a cluster with a large number of connections. Our experiment results demonstrate that RDMAvisor achieves high throughput for thousand of connections and maintains low CPU and memory overhead through adaptive RDMA transport selection.

Distributed, Parallel, and Cluster Computing,Networking and Internet Architecture,Performance
Toward Effective and Fair RDMA Resource Sharing.

Haonan Qiu,Xiaoliang Wang,Tianchen Jin,Zhuzhong Qian,Baoliu Ye,Bin Tang,Wenzhong Li,Sanglu Lu

DOI: https://doi.org/10.1145/3232565.3232636

2018-01-01

Abstract:Remote Direct Memory Access (RDMA) technique allows the messaging service that directly access the memory on remote machines, which provides low CPU overhead, low latency, and high throughput network transmission. On the other hand, however, due to the limited cache space in RDMA NIC (RNIC), it is still challenging to achieve effective and fair resource sharing across different applications. To address this problem, we present a scalable RDMA as a service to manage resource and deliver fair scheduling to applications' requests. We study the thread contention and preemptive schedule issues at end-hosts, and report the corresponding performance degradation through experiments. Then, we introduce Avatar, a model to manage memory and Queue Pairs (QPs) resource for a large number of connections, which eliminates the lock contention and provides fair data scheduling for applications with different priorities. Finally, we implement Avatar and demonstrate that Avatar can support a thousand of connections, improve the fairness and reduce the requests completion time up to 50% in comparison with the native RDMA.
SoftRDMA

Mao Miao,Fengyuan Ren,Xiaohui Luo,Jing Xie,Qingkai Meng,Wenxue Cheng

DOI: https://doi.org/10.1145/3106989.3106995

2017-01-01

Abstract:Recent academic and industrial work is exploring the challenges of using RDMA over Ethernet, to support highly reliable, latency-sensitive services in today's datacenters. Previous work on the high-speed packet I/O like netmap, DPDK, etc., and high-performance user-level stacks like mTCP, IX etc., rekindles our inspirations to implement a high-performance software RDMA over commodity Ethernet devices.
A Fast RDMA Offload Method for Unreliable Interconnection Networks

Shao-gang WANG,Wei-xia XU,Dan WU,Zheng-bin PANG,Jun XIA

DOI: https://doi.org/10.3969/j.issn.1674-2974.2015.08.016

2015-01-01

Abstract:Large data RDMA (Remote Data Memory Access)transport is the most commonly used par-allel communication mode for parallel computers,which has great impact on the whole system perform-ance.As the system size increases,the fault-tolerate architecture design faces new challenges.The inter-connection network usually uses the adaptive routing mode and becomes more unreliable.This paper pro-posed a fast RDMA offload method for unreliable interconnection networks,which can be efficiently imple-mented on the NIC hardware and provides reliable RDMA communication for upper driver and programs. Compared with the traditional approaches,the hardware overhead is greatly reduced.Another benefit is that it can partially retransmit the fault data,which greatly reduces the whole RDMA delay.Simulation results show that the RDMA delay is greatly reduced,compared with the traditional methods.
An efficient cloud-based elastic RDMA protocol for HPC applications

Hang Cao,Cheng Xu,Yunqi Han,Muhui Lin,Kai Shen,Geng Wang,Jinhu Li,Xiangzheng Sun,Ronghui He,Liang You,Hang Yang,Xiantao Zhang

DOI: https://doi.org/10.1007/s42514-023-00170-y

2023-09-15

CCF Transactions on High Performance Computing

Abstract:High-performance computing (HPC) networking is of great importance in scaling many HPC applications across multiple nodes. Generally, most HPC applications deployed on traditional supercomputers or clusters adopt RDMA protocols such as InfiniBand for inter-node networking to mitigate high latency during constant communication. As cloud-based HPC continues to emerge as a significant trend, utilizing RDMA in the cloud has become a challenging problem. To address this problem, We propose an efficient elastic RDMA Protocol (eRDMA) to enabling RDMA’s merits for HPC applications in the cloud. eRDMA applys the direct data movement (DDM) of cloud infrastructure processing Unit (CIPU), overlay of virtual private cloud (VPC), and compatibility for RDMA verbs to fully utilize the elastic resources with the features of RDMA network for HPC scenarios in the cloud. The effectiveness of eRDMA is demonstrated by various experimental results across different platforms for many HPC and general TCP applications.
POSTER: CAVER: Enhancing RDMA Load Balancing by Hunting Less-Congested Paths

Haotian Deng,Yuan Yang,Menghao Zhang,Mingwei Xu

DOI: https://doi.org/10.1145/3672202.3673729

2024-01-01

Abstract:Remote Direct Memory Access (RDMA) has become a prevailing technology for modern data centers (DCs) to achieve high throughput and low latency [8]. Many DCs have adopted RDMA over Converged Ethernet v2 (RoCEv2) [3] to provide superior performance for emerging application paradigms such as cloud storage [4] and distributed deep learning [11]. Network load balancing (LB) plays a critical role in optimizing the DC network performance. There is a large body of literature studying LB for traditional DCs [1, 2, 6, 7, 9], as it is well-known that the widely-used ECMP [12] has a limited LB performance. However, RDMA operates in a different manner compared to traditional TCP-based data transmission, and existing studies for traditional DCs do not well fit RDMA-enabled DCs. For example, RDMA is very sensitive to out-of-order packets which may lead to significant throughput degradation, and also, RDMA flow can hardly be partitioned into flowlets. Thus, existing packet-level LB approaches [6, 7] and flowlet-level LB approaches [2] perform poorly in RDMA-enabled DCs.
Efficient RDMA Communication Protocols

Konstantin Taranov,Fabian Fischer,Torsten Hoefler

DOI: https://doi.org/10.48550/arXiv.2212.09134

2022-12-21

Abstract:Developers of networked systems often work with low-level RDMA libraries to tailor network modules to take full advantage of offload capabilities offered by RDMA-capable network controllers. Because of the huge design space of networked data access protocols and variability in capabilities of RDMA infrastructure, developers tend to reinvent and reimplement common data exchange protocols, wasting months of development yet missing various performance and system capabilities. In this work, we summarise and categorize RDMA data exchange protocols and elaborate on what features they can offer to networked systems and what implications they have on their memory and network management.

Networking and Internet Architecture,Distributed, Parallel, and Cluster Computing
Maximizing the Benefit of RDMA at End Hosts

Xiaoliang Wang,Hexiang Song,Cam-Tu Nguyen,Dongxu Cheng,Tiancheng Jin

DOI: https://doi.org/10.1109/infocom42981.2021.9488875

2021-01-01

Abstract:RDMA is increasingly deployed in data center to meet the demands of ultra-low latency, high throughput and low CPU overhead. However, it is not easy to migrate existing applications from the TCP/IP stack to the RDMA. The developers usually need to carefully select communication primitives and manually tune the parameters for each single-purpose system. After operating the high-speed RDMA network, we identify multiple hidden costs which may cause degraded and/or unpredictable performance of RDMA-based applications. We demonstrate these hidden costs including the combination of complicated parameter settings, scalability of Reliable Connections, two-sided memory management and page alignment, resource contention among diverse traffics, etc. Furthermore, to address these problems, we introduce Nem, a suite that allows developers to maximize the benefit of RDMA by i) eliminating the resource contention at NIC cache through asynchronous resource sharing; ii) introducing hybrid page management based on messages sizes; iii) isolating flows of different traffic classes based hardware features. We implement the prototype of Nem and verify its effectiveness by rebuilding the RPC message service, which demonstrates the high throughput for large messages, low latency for small messages without compromising the low CPU utilization and good scalability performance for a large number of active connections.
RB2: Narrow the Gap Between RDMA Abstraction and Performance Via a Middle Layer

Haifeng Sun,Yixuan Tan,Yongtong Wu,Jiaqi Zhu,Qun Huang,Xin Yao,Gong Zhang

DOI: https://doi.org/10.1109/infocom52122.2024.10621169

2024-01-01

Abstract:Although the native RDMA interface allows for high throughput and low latency, its low-level abstraction raises significant programming challenges. Consequently, numerous systems encapsulate the RDMA interface into more user-friendly high-level abstractions such as Socket, MPI, and RPC. However, this ease of development often incurs considerable performance degradation. To address this trade-off, this paper introduces RB2, a high-performance RDMA-based Distributed Ring Buffer (DRB). RB2 serves as a middle layer that effectively conceals the low-level details of the RDMA interface while also facilitating extension to other high-level abstractions. Nonetheless, it is non-trivial for DRBs to preserve the RDMA performance. We optimize the performance of RB2 in three aspects. First, we perform micro-benchmarks to identify the pointer synchronization methods that are seemingly counter-intuitive but offer optimal performance improvements. Second, we propose an adaptive batching mechanism to alleviate the limitations of conventional fixed batching. Finally, we build an efficient memory subsystem using various optimization techniques. RB2 outperforms SOTA designs by achieving 2.5x to 7.5x throughput while maintaining comparable tail latency for small messages.
Survey on RDMA Protocol Applications and Security Protection Technologies

LIU Yumeng,TANG Zhengliang,LU Songfeng,ZHU Jianxin,LIU Yunqu

DOI: https://doi.org/10.11959/j.issn.2096-109x.2024022

2024-01-01

Abstract:The remote direct memory access(RDMA)technology,which has been effectively utilized to enhance data transfer rates and reduce CPU utilization,has played a significant role in various domains such as cross-regional data center transfers,high-performance computing,and rapid data read/write operations.However,despite its emerging status,RDMA has been lacking in widely recognized security solutions.With the expansion of RDMA applications from dedicated network environments to general Ethernet networks,the need to address the security risks faced by these applications has become imperative.A set of protective measures were established to ensure security without compromising transmission efficiency.Furthermore,the unique underlying implementation and protocol design of RDMA technology have resulted in incompatibility with existing mature security solutions and have led to the confrontation with specific security risks.The development of RDMA attack and defense technologies and the provision of security guarantees for the forthcoming widespread application of RDMA technology were elucidated The principles of RDMA technology and its implementation were proposed,with examples drawn from the InfiniBand(IB)and the RDMA over converged Ethernet(RoCE)second edition protocols.The security risks faced by RDMA applications in relevant scenarios were investigated,and a summary of the research progress in the RDMA security field over recent years was provided.Effective security solutions addressing these risks were compiled and,after their defensive capabilities were proven,a comparison of their advantages and disadvantages was conducted through theoretical analysis and experimental data.Finally,improvement plans and prospects for technological optimization in the RDMA attack and defense field were proposed.
SeqBalance: Congestion-Aware Load Balancing with no Reordering for RoCE

Huimin Luo,Jiao Zhang,Mingxuan Yu,Yongchen Pan,Tian Pan,Tao Huang

2024-07-13

Abstract:Remote Direct Memory Access (RDMA) is widely used in data center networks because of its high performance. However, due to the characteristics of RDMA's retransmission strategy and the traffic mode of AI training, current load balancing schemes for data center networks are unsuitable for RDMA. In this paper, we propose SeqBalance, a load balancing framework designed for RDMA. SeqBalance implements fine-grained load balancing for RDMA through a reasonable design and does not cause reordering problems. SeqBalance's designs are all based on existing commercial RNICs and commercial programmable switches, so they are compatible with existing data center networks. We have implemented SeqBalance in Mellanox CX-6 RNICs and Tofino switches. The results of hardware testbed experiments and large-scale simulations show that compared with existing load balancing schemes, SeqBalance improves 18.7% and 33.2% on average FCT and 99th percentile FCT.

Networking and Internet Architecture
X-RDMA: Effective RDMA Middleware in Large-scale Production Environments.

Teng Ma,Tao Ma,Zhuo Song,Jingxuan Li,Huaixin Chang,Kang Chen,Hai Jiang,Yongwei Wu

DOI: https://doi.org/10.1109/cluster.2019.8891004

2019-01-01

Abstract:X-RDMA is a communication middleware deployed and heavily used in Alibaba's large-scale cluster hosting cloud storage and database systems. Unlike recent research projects which purely focus on squeezing out the raw hardware performance, it puts emphasis on robustness, scalability and maintainability of large-scale production clusters. X-RDMA integrates necessary features, not available in current RDMA ecosystem, to release the developers from complex and imperfect details. X-RDMA simplifies the programming model, extends RDMA protocols for application awareness, and proposes mechanisms for resource management with thousands of connections per machine. It also reduces the work for administration and performance tuning with built-in tracing, tuning and monitoring tools. X-RDMA has been deployed in several large-scale clusters with over 4000 servers in Alibaba cloud since 2016. It can save at least 70% development and maintenance time over RDMA, effectively improve performance and reduce network jitter especially when production servers are under pressure. It also helped locate over 30 issues in different layers of productions with over 5000 connections for each server on average.
RDMA Load Balancing via Data Partition

Yi Wang,Ya-nan Jiang,Qiufang Ma,Chen Tian,Bo Bai,Gong Zhang

DOI: https://doi.org/10.1109/ICCCN.2019.8847077

2019-01-01

Abstract:With the development of data center networks, traditional TCP/IP cannot support the demand in data centers. Remote Direct Memory Access (RDMA) technology could improve the performance of DCN significantly because of high throughput and low latency. However, load balancing is a key issue in RDMA which has not been solved distribute. This paper will propose an algorithm to solve the load balance problem in RDMA on application layer without hardware changing. The main idea is to divide data to chunks and data chunks on multiple reachable paths for transmission. However, it is no trivial to find the optimal chunk size and the path number, some empirical values are found by varieties of experiments and tests. Moreover, the chunk allocation scheme also needs to consider the traffic condition in DCNs to find more free paths to transmit. We evaluate the algorithm in application layer with ns3 simulator. The experiment results show that with our algorithm the completion time can decrease 81.03% at most.
RDMA is Turing complete, we just did not know it yet!

Waleed Reda,Marco Canini,Dejan Kostić,Simon Peter

DOI: https://doi.org/10.48550/arXiv.2103.13351

2021-11-17

Abstract:It is becoming increasingly popular for distributed systems to exploit offload to reduce load on the CPU. Remote Direct Memory Access (RDMA) offload, in particular, has become popular. However, RDMA still requires CPU intervention for complex offloads that go beyond simple remote memory access. As such, the offload potential is limited and RDMA-based systems usually have to work around such limitations. We present RedN, a principled, practical approach to implementing complex RDMA offloads, without requiring any hardware modifications. Using self-modifying RDMA chains, we lift the existing RDMA verbs interface to a Turing complete set of programming abstractions. We explore what is possible in terms of offload complexity and performance with a commodity RDMA NIC. We show how to integrate these RDMA chains into applications, such as the Memcached key-value store, allowing us to offload complex tasks such as key lookups. RedN can reduce the latency of key-value get operations by up to 2.6x compared to state-of-the-art KV designs that use one-sided RDMA primitives (e.g., FaRM-KV), as well as traditional RPC-over-RDMA approaches. Moreover, compared to these baselines, RedN provides performance isolation and, in the presence of contention, can reduce latency by up to 35x while providing applications with failure resiliency to OS and process crashes.

Distributed, Parallel, and Cluster Computing
A Comprehensive Evaluation of RDMA-enabled Concurrency Control Protocols.

Chao Wang,Kezhao Huang,Xuehai Qian

2020-01-01

Abstract:On-line transaction processing (OLTP) applications require efficient distributed transaction execution. When a transaction accesses multiple records in remote machines, network performance is a crucial factor affecting transaction latency and throughput. Due to its high bandwidth and very low latency, RDMA (Remote Direct Memory Access) has achieved much higher performance for distributed transactions than traditional TCP-based systems. RDMA provides primitives for both two-sided and one-sided communication. Although recent works have intensively studied the benefits of RDMA in distributed transaction systems, they either focus on primitive-level comparisons of two communication models (one-sided vs. two-sided) or only study one concurrency control protocol. A comprehensive understanding of the implication of RDMA for various concurrency control protocols is an open problem. In this paper, we build RCC, the first unified and comprehensive RDMA-enabled distributed transaction processing framework supporting six concurrency control protocols using either two-sided or one-sided primitives. We intensively optimize the performance of each protocol without bias, using known techniques such as co-routines, outstanding requests, and doorbell batching. Based on RCC, we conduct the first and most comprehensive (to the best of our knowledge) study of the six representative distributed concurrency control protocols on two clusters with different RDMA network capabilities.
DySched: Relieving Large-Scale Incast for Cloud-Native RDMA Applications

Jiejian Wu,Zhe Wang,Teng Ma,Linghe Kong,Yutong Liu,Zhuo Song,Jiadi Yu,Yong Yang,Tao Ma,Guihai Chen

DOI: https://doi.org/10.1109/ispa-bdcloud-socialcom-sustaincom59178.2023.00035

2023-01-01

Abstract:Remote Direct Memory Access (RDMA), with its advantages of large throughput and low latency, has been widely deployed for applications like cloud storage and distributed machine-learning training. However, in large-scale RDMA networks, congestion caused by incast often leads to severe performance degradation of throughput and latency. Different from the traditional data centers, the novel cloud-native infrastructure brings new challenges to resolve the incast issues: the underlying network fabric/configurations are managed by the cloud vendors and the cloud-native applications running on the virtualized/containerized platform are agnostic to the network.We design DySched, a novel middleware to relieve the largescale incast issues of RDMA applications, without any dependencies on network fabric/configuration (topology, flow control, congestion notification) in a cloud-native environment. DySched creatively utilizes the inherent time cost of posting RDMA requests to determine the inflight data allocation of each RDMA connection. DySched thus achieves an ideal full bandwidth utilization and no congestion at the receiver side. DySched has been deployed in two typical clusters in Alibaba Cloud. Results show that DySched improves throughput by 22% to 60% and reduces job completion time by 4% to 67% compared to state-of-the-art methods.
DTS: A Dual Transport Switching Scheme for RDMA-based Applications

Yuxin Chen,Dongyang Wang,Gang Lu,Junhong Ye,Feng Jin,Zhiqiang He,Bei Hua

DOI: https://doi.org/10.1109/ipccc55026.2022.9894343

2022-01-01

Abstract:RDMA has been widely adopted by distributed applications for its low latency, high throughput, and near-zero CPU overhead. However, high availability is still lacked for RDMA-based applications, especially in the cases of network failure or NIC upgrade. In this paper, we propose DTS, a scheme for RDMA-based applications to provide high availability by adopting the active and standby transports and switching them transparently and fast when necessary. We implement the prototype of DTS based on the Libfabric RxD provider. Our evaluation demonstrates that with DTS existing RDMA-based applications can be deployed with zero downtime and the performance degradation is ignorable.

Scalable and Reliable RDMA.

Scalable RDMA RPC on Reliable Connection with Efficient Resource Sharing

Optimizing Distributed Systems with Remote Direct Memory Access

MC-RDMA: Improving Replication Performance of RDMA-based Distributed Systems with Reliable Multicast Support

RDMAvisor: Toward Deploying Scalable and Simple RDMA as a Service in Datacenters

Toward Effective and Fair RDMA Resource Sharing.

SoftRDMA

A Fast RDMA Offload Method for Unreliable Interconnection Networks

An efficient cloud-based elastic RDMA protocol for HPC applications

POSTER: CAVER: Enhancing RDMA Load Balancing by Hunting Less-Congested Paths

Efficient RDMA Communication Protocols

Maximizing the Benefit of RDMA at End Hosts

RB2: Narrow the Gap Between RDMA Abstraction and Performance Via a Middle Layer

Survey on RDMA Protocol Applications and Security Protection Technologies

SeqBalance: Congestion-Aware Load Balancing with no Reordering for RoCE

X-RDMA: Effective RDMA Middleware in Large-scale Production Environments.

RDMA Load Balancing via Data Partition

RDMA is Turing complete, we just did not know it yet!

A Comprehensive Evaluation of RDMA-enabled Concurrency Control Protocols.

DySched: Relieving Large-Scale Incast for Cloud-Native RDMA Applications

DTS: A Dual Transport Switching Scheme for RDMA-based Applications