Abstract:The high computational and memory requirements of generative large language models (LLMs) make it challenging to serve them cheaply. This paper aims to reduce the monetary cost for serving LLMs by leveraging preemptible GPU instances on modern clouds, which offer accesses to spare GPUs at a much cheaper price than regular instances but may be preempted by the cloud at any time. Serving LLMs on preemptible instances requires addressing challenges induced by frequent instance preemptions and the necessity of migrating instances to handle these preemptions. This paper presents SpotServe, the first distributed LLM serving system on preemptible instances. Several key techniques in SpotServe realize fast and reliable serving of generative LLMs on cheap preemptible instances. First, SpotServe dynamically adapts the LLM parallelization configuration for dynamic instance availability and fluctuating workload, while balancing the trade-off among the overall throughput, inference latency and monetary costs. Second, to minimize the cost of migrating instances for dynamic reparallelization, the task of migrating instances is formulated as a bipartite graph matching problem, which uses the Kuhn-Munkres algorithm to identify an optimal migration plan that minimizes communications. Finally, to take advantage of the grace period offered by modern clouds, we introduce stateful inference recovery, a new inference mechanism that commits inference progress at a much finer granularity and allows SpotServe to cheaply resume inference upon preemption. We evaluate on real spot instance preemption traces and various popular LLMs and show that SpotServe can reduce the P99 tail latency by 2.4 - 9.1x compared with the best existing LLM serving systems. We also show that SpotServe can leverage the price advantage of preemptive instances, saving 54 compared with only using on-demand instances.

SMILE: A Cost-Effective System for Serving Massive Pretrained Language Models in the Cloud.

GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost

Efficient Deployment of Large Language Model Across Cloud-Device Systems

CPM-2: Large-scale Cost-effective Pre-trained Language Models

COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Models

CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaboration

BlockLLM: Multi-tenant Finer-grained Serving for Large Language Models

LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMs

MLModelCI: An Automatic Cloud Platform for Efficient MLaaS

SpotServe: Serving Generative Large Language Models on Preemptible Instances

LLMs as On-demand Customizable Service

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

Improving Large Models with Small models: Lower Costs and Better Performance

Inference Performance Optimization for Large Language Models on CPUs

Cloud-Device Collaborative Learning for Multimodal Large Language Models

An Empirical Analysis and Resource Footprint Study of Deploying Large Language Models on Edge Devices

ELMS: Elasticized Large Language Models On Mobile Devices

Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity

RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs

P/D-Serve: Serving Disaggregated Large Language Model at Scale

Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistance