Understanding the Benefits of Hardware-Accelerated Communication in Model-Serving Applications

Walid A. Hanafy,Limin Wang,Hyunseok Chang,Sarit Mukherjee,T. V. Lakshman,Prashant Shenoy

2023-07-10

Abstract:It is commonly assumed that the end-to-end networking performance of edge offloading is purely dictated by that of the network connectivity between end devices and edge computing facilities, where ongoing innovation in 5G/6G networking can help. However, with the growing complexity of edge-offloaded computation and dynamic load balancing requirements, an offloaded task often goes through a multi-stage pipeline that spans across multiple compute nodes and proxies interconnected via a dedicated network fabric within a given edge computing facility. As the latest hardware-accelerated transport technologies such as RDMA and GPUDirect RDMA are adopted to build such network fabric, there is a need for good understanding of the full potential of these technologies in the context of computation offload and the effect of different factors such as GPU scheduling and characteristics of computation on the net performance gain achievable by these technologies. This paper unveils detailed insights into the latency overhead in typical machine learning (ML)-based computation pipelines and analyzes the potential benefits of adopting hardware-accelerated communication. To this end, we build a model-serving framework that supports various communication mechanisms. Using the framework, we identify performance bottlenecks in state-of-the-art model-serving pipelines and show how hardware-accelerated communication can alleviate them. For example, we show that GPUDirect RDMA can save 15--50\% of model-serving latency, which amounts to 70--160 ms.

Performance

What problem does this paper attempt to address?

The paper primarily explores the performance improvements that hardware-accelerated communication technologies (such as RDMA and GPUDirect RDMA) can bring to model service applications in edge computing. Specifically, the research aims to address the following issues:

1. **Understanding the potential of hardware-accelerated communication technologies**: The study seeks to reveal the extent of performance improvements that the latest hardware-accelerated communication technologies (such as RDMA and GPUDirect RDMA) can bring to model services in an edge computing environment. It also analyzes the impact of various factors (such as GPU scheduling, computation type, and scale) on the overall performance gains.

2. **Identifying bottlenecks in the model service pipeline**: By constructing a model service framework that supports multiple communication mechanisms (such as TCP, RDMA, etc.), researchers can identify performance bottlenecks in modern model service pipelines and demonstrate how hardware-accelerated communication can alleviate these issues.

3. **Evaluating the benefits of hardware-accelerated communication technologies**: The research demonstrates through experiments that GPUDirect RDMA can significantly reduce model service latency (saving 15%-50%, i.e., 70-160 milliseconds) and further analyzes the importance of the proportion of time spent on communication in the model service pipeline for performance improvement.

In summary, the main objective of this paper is to explore and quantify the potential performance improvements brought by using hardware-accelerated communication technologies (especially GPUDirect RDMA) for model services in edge computing scenarios. It also aims to reveal the advantages and limitations of these technologies through empirical research.