Abstract:In the context of Machine Learning as a Service (MLaaS) clouds, the extensive use of Large Language Models (LLMs) often requires efficient management of significant query loads. When providing real-time inference services, several challenges arise. Firstly, increasing the number of GPUs may lead to a decrease in inference speed due to heightened communication overhead, while an inadequate number of GPUs can lead to out-of-memory errors. Secondly, different deployment strategies need to be evaluated to guarantee optimal utilization and minimal inference latency. Lastly, inefficient orchestration of inference queries can easily lead to significant Service Level Objective (SLO) violations. Lastly, inefficient orchestration of inference queries can easily lead to significant Service Level Objective (SLO) violations. To address these challenges, we propose a Unified and Efficient approach for Large Language Model inference serving (UELLM), which consists of three main components: 1) resource profiler, 2) batch scheduler, and 3) LLM deployer. UELLM minimizes resource overhead, reduces inference latency, and lowers SLO violation rates. Compared with state-of-the-art (SOTA) techniques, UELLM reduces the inference latency by 72.3% to 90.3%, enhances GPU utilization by 1.2X to 4.1X, and increases throughput by 1.92X to 4.98X, it can also serve without violating the inference latency SLO.

What problem does this paper attempt to address?

The paper attempts to address several key challenges faced by large language model (LLM) inference services in a Machine Learning as a Service (MLaaS) cloud environment: 1. **Increasing the number of GPUs may lead to slower inference speeds**: Due to increased communication overhead, simply adding more GPUs may result in slower inference speeds; insufficient GPUs may lead to memory overflow errors. 2. **Different deployment strategies need to be evaluated**: To ensure optimal resource utilization and minimal inference latency, different deployment strategies need to be evaluated. 3. **Ineffective inference request scheduling can lead to significant Service Level Objective (SLO) violations**: Unreasonable request scheduling can easily lead to SLO violations, affecting service quality. To address these challenges, the paper proposes a unified and efficient large language model inference service framework (UELLM), which consists of three main components: 1. **Resource Profiler**: Characterizes resource usage by predicting the resource requirements of each inference request. 2. **Batch Scheduler**: Effectively batches requests based on the predictions of the Resource Profiler to reduce inference latency and meet SLO requirements. 3. **LLM Deployer**: Efficiently deploys LLMs based on the current cluster hardware topology and LLM characteristics to improve resource utilization and reduce resource overhead. UELLM aims to minimize resource overhead, reduce inference latency, and lower the SLO violation rate. Compared to existing state-of-the-art technologies, UELLM can reduce inference latency by 72.3% to 90.3%, increase GPU utilization by 1.2 times to 4.1 times, increase throughput by 1.92 times to 4.98 times, and provide services without violating inference latency SLOs.

UELLM: A Unified and Efficient Approach for LLM Inference Serving

Efficient Deployment of Large Language Model Across Cloud-Device Systems

Large Language Models (llms) Inference Offloading and Resource Allocation in Cloud-Edge Networks: an Active Inference Approach

Fast Distributed Inference Serving for Large Language Models

PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Services

SplitLLM: Collaborative Inference of LLMs for Model Placement and Throughput Optimization

ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency

Llumnix: Dynamic Scheduling for Large Language Model Serving

Efficient and Economic Large Language Model Inference with Attention Offloading

A System for Microserving of LLMs

ServerlessLLM: Low-Latency Serverless Inference for Large Language Models

AcceLLM: Accelerating LLM Inference using Redundancy for Load Balancing and Data Locality

LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services

GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments

Edge-LLM: A Collaborative Framework for Large Language Model Serving in Edge Computing

ELMS: Elasticized Large Language Models On Mobile Devices

EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism

Efficient LLM Scheduling by Learning to Rank

Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference

Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems