Abstract:Efficient inference in Large Language Models (LLMs) is impeded by the growing memory demands of key-value (KV) caching, especially for longer sequences. Traditional KV cache eviction strategies, which prioritize less critical KV-pairs based on attention scores, often degrade generation quality, leading to issues such as context loss or hallucinations. To address this, we introduce Dynamic Discriminative Operations (D2O), a novel method that utilizes two-level discriminative strategies to optimize KV cache size without fine-tuning, while preserving essential context. Initially, by observing varying densities of attention weights between shallow and deep layers, we use this insight to determine which layers should avoid excessive eviction to minimize information loss. Subsequently, for the eviction strategy in each layer, D2O innovatively incorporates a compensation mechanism that maintains a similarity threshold to re-discriminate the importance of previously discarded tokens, determining whether they should be recalled and merged with similar tokens. Our approach not only achieves significant memory savings and enhances inference throughput by more than 3 times but also maintains high-quality long-text generation. Extensive experiments across various benchmarks and LLM architectures have demonstrated that D2O significantly enhances performance with a constrained KV cache budget.

What problem does this paper attempt to address?

This paper attempts to address the memory requirements that large - language models (LLMs) face when processing long texts, especially the requirements for key - value (KV) caches. As the sequence length increases, the memory consumption of the KV cache rises sharply, which not only limits the model's inference efficiency but may also lead to problems such as context loss or hallucination. Although existing KV cache eviction strategies can reduce memory usage to a certain extent, they often sacrifice generation quality and result in information loss. To this end, the paper proposes **Dynamic Discriminative Operations (D2O)**, a new method that optimizes the KV cache size through a two - layer discriminative strategy while maintaining the integrity of the context. Specifically: 1. **Layer - level Discriminative Operations**: D2O observes the differences in the density of shallow and deep attention weights and uses this finding to determine which layers should avoid excessive eviction to minimize information loss. For example, the attention maps in the shallow layers are usually dense, while those in the deep layers show a step - like sparse pattern. Based on this, D2O adopts different eviction ratios for different layers, thus managing memory more effectively. 2. **Token - level Discriminative Operations**: For the eviction strategy of each layer, D2O introduces a compensation mechanism. By maintaining a similarity threshold, it re - evaluates the importance of previously discarded tokens and decides whether to recall them and merge them with similar tokens. This method not only reduces information loss but also improves the throughput of inference. Through these strategies, D2O not only significantly saves memory but also increases the inference throughput by more than three times while maintaining the high - quality long - text generation ability. Experimental results show that D2O exhibits significant performance improvements in various benchmark tests and LLM architectures.

D2O: Dynamic Discriminative Operations for Efficient Generative Inference of Large Language Models

InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity

Efficient LLM Inference with Kcache

Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference

NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time

XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference

In-context KV-Cache Eviction for LLMs via Attention-Gate

Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention

CORM: Cache Optimization with Recent Message for Large Language Model Inference

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

LoCoCo: Dropping In Convolutions for Long Context Compression

Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs

Anchor Attention, Small Cache: Code Generation with Large Language Models

Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference

DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

Sequence Can Secretly Tell You What to Discard.

Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache

Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks