Abstract:A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation. However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference. Our approach is based on the observation that KV cache states exhibit high similarity between the adjacent layers in the middle-to-deep portion of LLMs. To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead. Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity. We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with 4-bit MiniCache achieves a remarkable compression ratio of up to 5.02x, enhances inference throughput by approximately 5x, and reduces the memory footprint by 41% compared to the FP16 full cache baseline, all while maintaining near-lossless performance.

Layer-Condensed KV Cache for Efficient Inference of Large Language Models

A Method for Building Large Language Models with Predefined KV Cache Capacity

A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference

XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference

KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing

LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management

LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy

PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference

PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation

Unifying KV Cache Compression for Large Language Models with LeanKV

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

CORM: Cache Optimization with Recent Message for Large Language Model Inference

SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction

MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

Reducing Transformer Key-Value Cache Size with Cross-Layer Attention

A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity

Lossless KV Cache Compression to 2%

MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding

Efficient LLM Inference with Kcache