Abstract:We survey the large language model (LLM) serving area to understand the intricate dynamics between cost-efficiency and accuracy, which is magnified by the growing need for longer contextual understanding when deploying models at a massive scale. Our findings reveal that works in this space optimize along three distinct but conflicting goals: improving serving context length (C), improving serving accuracy (A), and improving serving performance (P). Drawing inspiration from the CAP theorem in databases, we propose a CAP principle for LLM serving, which suggests that any optimization can improve at most two of these three goals simultaneously. Our survey categorizes existing works within this framework. We find the definition and continuity of user-perceived measurement metrics are crucial in determining whether a goal has been met, akin to prior CAP databases in the wild. We recognize the CAP principle for LLM serving as a guiding principle, rather than a formal theorem, to inform designers of the inherent and dynamic trade-offs in serving models. As serving accuracy and performance have been extensively studied, this survey focuses on works that extend serving context length and address the resulting challenges.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to explore the complex and dynamic relationship between cost - effectiveness and accuracy in the deployment of large - scale language models (LLMs), especially when these models need to handle longer contexts. As the demand for understanding longer contexts continues to grow, optimizing the performance of models in large - scale deployment becomes particularly important. Through investigation, the authors found that in the field of LLM services, research work mainly focuses on optimizing three conflicting goals: 1. **Increasing the service context length (C)**: that is, increasing the length of the context that the model can effectively handle. 2. **Improving the service accuracy (A)**: that is, improving the precision of the model's output. 3. **Enhancing the service performance (P)**: that is, improving the speed and efficiency of the model's inference. The paper proposes an **LLM Service CAP Principle** similar to the database CAP theorem, stating that any optimization measure can at most simultaneously improve two of the above three goals. Specifically: - **C (Context)**: refers to the length of the context effectively processed as perceived by the user. - **A (Accuracy)**: refers to the output precision evaluated based on specific task metrics. - **P (Performance)**: refers to the efficiency of token processing and generation as perceived by the user. According to this principle, the authors classify existing research work and emphasize that the definition and continuity of user - perceived measurement indicators are crucial for determining whether a goal has been achieved. In addition, the paper also pays special attention to the work of extending the context length and the challenges it brings, because this places higher requirements on balancing service accuracy and performance. ### Main contributions 1. **Proposing the CAP principle for LLM services**: Mapping existing work into the CAP framework and highlighting the tensions between them. 2. **Overall perspective**: Starting from the entire large - scale LLM service system rather than focusing on a specific technology or layer. In this way, the paper provides guidance for designers to help them understand the inherent and dynamic trade - offs faced when deploying large - scale systems.

The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving

ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency

Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems

A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

Densing Law of LLMs

LLM Inference Serving: Survey of Recent Advances and Opportunities

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism

Through the Lens of Core Competency: Survey on Evaluation of Large Language Models

A Survey on Human-Centric LLMs

OptLLM: Optimal Assignment of Queries to Large Language Models

Enhancing Large Language Models' Situated Faithfulness to External Contexts

Efficient Large Language Models: A Survey

Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models

Understanding User Experience in Large Language Model Interactions

A Survey of Large Language Models

L-Eval: Instituting Standardized Evaluation for Long Context Language Models

The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey

FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition

Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models

Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache