Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction

Haoran Qiu,Weichao Mao,Archit Patke,Shengkun Cui,Saurabh Jha,Chen Wang,Hubertus Franke,Zbigniew T. Kalbarczyk,Tamer Başar,Ravishankar K. Iyer

2024-04-12

Abstract:Large language models (LLMs) have been driving a new wave of interactive AI applications across numerous domains. However, efficiently serving LLM inference requests is challenging due to their unpredictable execution times originating from the autoregressive nature of generative models. Existing LLM serving systems exploit first-come-first-serve (FCFS) scheduling, suffering from head-of-line blocking issues. To address the non-deterministic nature of LLMs and enable efficient interactive LLM serving, we present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths. Our open-source SSJF implementation does not require changes to memory management or batching strategies. Evaluations on real-world datasets and production workload traces show that SSJF reduces average job completion times by 30.5-39.6% and increases throughput by 2.2-3.6x compared to FCFS schedulers, across no batching, dynamic batching, and continuous batching settings.

Distributed, Parallel, and Cluster Computing,Computation and Language,Machine Learning

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily addresses the issue of efficient inference services for large language models (LLMs) in interactive applications. Specifically, due to the autoregressive nature of LLMs, the length of the output sequence is unpredictable, leading to uncertain execution times for inference requests. Most existing LLM service systems adopt a First-Come-First-Serve (FCFS) scheduling strategy, which can cause queue blocking issues and negatively impact user experience.

The paper proposes a Speculative Shortest-Job-First (SSJF) scheduler, which uses a lightweight proxy model (e.g., a fine-tuned BERT-base model) to predict the output sequence length of the LLM, thereby optimizing the scheduling strategy. SSJF can significantly reduce the average task completion time and increase throughput without requiring any changes to memory management and batching strategies.

### Main Contributions

1. **Speculative Request Scheduler**: Proposes a scheduler based on a lightweight proxy model (i.e., SSJF) to address the non-deterministic nature of generative LLMs.
2. **Open Source Implementation and Evaluation**: Provides an open-source implementation of SSJF and evaluates it on real-world datasets and production workloads.
3. **Other Potential Uses**: Discusses other potential uses of the proxy model in LLM services beyond request scheduling, such as dynamic memory allocation and cache strategy optimization.