Abstract:Serving numerous users and requests concurrently requires good fairness in Large Language Models (LLMs) serving system. This ensures that, at the same cost, the system can meet the Service Level Objectives (SLOs) of more users , such as time to first token (TTFT) and time between tokens (TBT), rather than allowing a few users to experience performance far exceeding the SLOs. To achieve better fairness, the preemption-based scheduling policy dynamically adjusts the priority of each request to maintain balance during runtime. However, existing systems tend to overly prioritize throughput, overlooking the overhead caused by preemption-induced context switching, which is crucial for maintaining fairness through priority adjustments. In this work, we identify three main challenges that result in this overhead. 1) Inadequate I/O utilization. 2) GPU idleness. 3) Unnecessary I/O transmission during multi-turn conversations. Our key insight is that the block-based KV cache memory policy in existing systems, while achieving near-zero memory waste, leads to discontinuity and insufficient granularity in the KV cache memory. To respond, we introduce FastSwitch, a fairness-aware serving system that not only aligns with existing KV cache memory allocation policy but also mitigates context switching overhead. Our evaluation shows that FastSwitch outperforms the state-of-the-art LLM serving system vLLM with speedups of 1.4-11.2x across different tail TTFT and TBT.

Fast Context Switching Schedule Algorithm

Fast Combination of Scheduling Chains under Resource and Time Constraints

Freshness-Aware Real-Time Scheduling Algorithm for Context Reasoning

An Efficient Fault-Tolerant Scheduling Algorithm for Periodic Real-Time Tasks in Heterogeneous Platforms

Global-optimized scheduling algorithm based on combination of sub-graphs

Cloud Scheduling Algorithm Based on Fuzzy Clustering

FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving

Simple and Fair Scheduling Algorithm for Combined Input-Crosspoint-Queued Switch

Share Memory Aware Scheduler

A Fair Service and Dynamic Round Robin scheduling scheme for CICQ switches

Priority Based Fair Scheduling : A Memory Scheduler Design for Chip-Multiprocessor Systems

Fuzzy logic based feedback scheduler for embedded control systems

A fast-flexible strategy based approach to solving employee scheduling problem considering soft work time

Efficient Coflow Scheduling in Hybrid-Switched Data Center Networks

Efficient Scheduling for Multi-Stage Coflows

Switching Architecture Based on Parallel Storage Scheme and Its Scheduling Algorithm

Task Scheduling for Heterogeneous Computing Based on Learning Classifier System

Fast TS Algorithm Based on Block Properties of FSP

Switching Serial-Number Coding Scheme and Its Application in FFS Scheduling Problem with Inter-stage Constraints

A Novel Heterogeneous Scheduling Algorithm with Improved Task Priority

Enhanced Parallel Application Scheduling Algorithm with Energy Consumption Constraint in Heterogeneous Distributed Systems