LTNoT: Realizing the Trade-Offs Between Latency and Throughput in NVMe over TCP.

Wenhao Gu,Xuchao Xie,Dezun Dong

DOI: https://doi.org/10.1007/978-3-031-22677-9_22

2022-01-01

Abstract:NVMe over Fabrics (NVMeoF) is a new emerging storage disaggregation protocol specially designed for datacenters with high-performance NVMe SSDs and interconnection networks. However, existing NVMeoF implementations cannot meet the differentiated I/O demands of the diverse applications running in datacenters. This is because the applications usually show significantly different I/O characteristics and requirements, e.g., some applications (L-apps) are sensitive to latency while other applications (T-apps) show high throughput demands to storage systems. When L-apps and T-apps access remote NVMe SSDs via a same NVMeoF storage network, the I/O requests issued from these applications are equally treated and handled following the same I/O path in state-of-the-art NVMeoF implementations. This will finally incur severe I/O interference between L-apps and T-apps. In this paper, we propose LTNoT, an end-to-end packet processing scheme with dedicated I/O pipelines for L-apps and T-apps in NVMe over TCP (NoT) implementation. Specifically, LTNoT separates T-apps and L-apps resources in each NVMeoF queue pair to achieve inter-queue I/O isolation, transfers capsule and data in batch along with the T-app pipeline to achieve interrupt-coalescing, and introduces immediate-delivery and workqueue-priority to optimize L-app request process. We implemented LTNoT in Linux Kernel and evaluated it using real-world benchmarks and applications. Our experimental results demonstrate that LTNoT can achieve 48.13% and 53.38% lower L-apps latency than i10 and NoT respectively, increase bandwidth by up to 33.31% than NoT on average, thus LTNoT can effectively alleviate the I/O interference issue in NVMe over TCP without introducing any negative performance impacts on either L-apps or T-apps.