Optimal Workload Placement on Multi-Instance GPUs

Bekir Turkkan,Pavankumar Murali,Pavithra Harsha,Rohan Arora,Gerard Vanloo,Chandra Narayanaswami

2024-09-11

Abstract:There is an urgent and pressing need to optimize usage of Graphical Processing Units (GPUs), which have arguably become one of the most expensive and sought after IT resources. To help with this goal, several of the current generation of GPUs support a partitioning feature, called Multi-Instance GPU (MIG) to allow multiple workloads to share a GPU, albeit with some constraints. In this paper we investigate how to optimize the placement of Large Language Model (LLM)-based AI Inferencing workloads on GPUs. We first identify and present several use cases that are encountered in practice that require workloads to be efficiently placed or migrated to other GPUs to make room for incoming workloads. The overarching goal is to use as few GPUs as possible and to further minimize memory and compute wastage on GPUs that are utilized. We have developed two approaches to address this problem: an optimization method and a heuristic method. We benchmark these with two workload scheduling heuristics for multiple use cases. Our results show up to 2.85x improvement in the number of GPUs used and up to 70% reduction in GPU wastage over baseline heuristics. We plan to enable the SRE community to leverage our proposed method in production environments.

Distributed, Parallel, and Cluster Computing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is optimizing the placement of large - language - model (LLM) inference workloads on multi - instance GPUs (MIG). Specifically, the authors explore how to efficiently arrange these workloads within limited GPU resources to reduce the number of required GPUs and minimize the waste of GPU memory and computing resources. The paper mentions that with the wide application of large - language models, GPU resources have become extremely precious and the demand far exceeds the supply, so optimizing their usage efficiency has become particularly important. In addition, the paper also focuses on how to achieve the optimal configuration of workloads through effective migration strategies, thereby improving the utilization and availability of GPUs while reducing the interruption time caused by migration.

To achieve the above goals, the paper proposes two methods to solve this problem:
1. **Optimization method**: A multi - objective global optimization method based on mixed - integer programming (MIP), which can handle optimization problems in all usage scenarios simultaneously.
2. **Heuristic method**: A fast rule - based heuristic method that optimizes each usage scenario separately.

Both of these methods were benchmarked in multiple usage scenarios, and the results show that compared with the baseline method, these methods can significantly reduce the number of required GPUs (up to 2.85 times improvement) and reduce GPU resource waste (up to 70% reduction). In addition, the paper also discusses the application prospects of these methods in actual production environments, aiming to provide effective tools for the system reliability engineer (SRE) community to implement these optimization strategies in production environments.