Understanding Delays in AF\_XDP-based Applications

Killian Castillon du Perron,Dino Lopez Pacheco,Fabrice Huet
2024-02-19
Abstract:Packet processing on Linux can be slow due to its complex network stack. To solve this problem, there are two main solutions: eXpress Data Path (XDP) and Data Plane Development Kit (DPDK). XDP and the AF XDP socket offer full interoperability with the legacy system and is being adopted by major internet players like Open vSwitch or Facebook. While the performance evaluation of AF XDP against the legacy protocol stack in the kernel or against DPDK has been studied in the literature, the impact of the multiple socket parameters and the system configuration on its latency has been left aside. To address this, we conduct an experimental study to understand the XDP/AF XDP ecosystem and detect microseconds delays to better architect future latency-sensitive applications. Since the performance of AF XDP depends on multiple parameters found in different layers, finding the configuration minimizing its latency is a challenging task. We rely on a classification algorithm to group the performance results, allowing us to easily identify parameters with the biggest impact on performance at different loads. Last, but not least, we show that some configurations can significantly decrease the benefits of AF XDP, leading to undesirable behaviors, while other configurations are able to reduce such round trip delays to an impressive value of 6.5 $\mu$s in the best case, including the tracing overhead. In summary, AF XDP is a promising solution, and careful selection of both application and socket parameters can significantly improve performance.
Networking and Internet Architecture
What problem does this paper attempt to address?
The paper primarily discusses the latency issues encountered when processing network packets using eXpress Data Path (XDP) and AF XDP socket technology in the Linux system. The author points out that due to the complex network stack of the Linux kernel, packet processing can become slow, and XDP and the Data Plane Development Kit (DPDK) are two main solutions to accelerate this process. Although previous studies have compared the performance of AF XDP with traditional protocol stacks or DPDK, there is a relative lack of in-depth research on the impact of various socket parameters and system configurations on latency. Through experimental research, the paper analyzes the characteristics of the XDP/AF XDP ecosystem and detects microsecond-level latency to optimize the architecture of future latency-sensitive applications. The study finds that the performance of AF XDP is affected by multiple levels of parameters, and finding the configuration that minimizes latency is a challenge. The author uses classification algorithms to group performance results, thereby identifying the parameters that have the most significant impact on performance under different loads. The research also shows that certain configurations may significantly reduce the benefits of AF XDP, leading to adverse behavior, while other configurations can reduce round-trip latency to an impressive 6.5 microseconds in the best case, including tracing overhead. The paper further details the architecture of XDP and AF XDP, as well as how to optimize performance by adjusting hardware, driver settings, and AF XDP socket parameters. Through experiments, the author generated nearly 400 different configurations and used classification algorithms to analyze the results, determining which parameters lead to the best and worst latency. Experimental results indicate that under the correct parameters, power-saving mechanisms have almost no impact on the latency of AF XDP sockets; and under the optimal combination of parameters, the round-trip latency between two servers can reach 6.5 microseconds. In summary, AF XDP is a promising solution, and carefully selecting application and socket parameters can significantly improve performance, which is especially important for modern cloud applications seeking to reduce latency to the microsecond level.