Abstract:A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation. However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference. Our approach is based on the observation that KV cache states exhibit high similarity between the adjacent layers in the middle-to-deep portion of LLMs. To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead. Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity. We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with 4-bit MiniCache achieves a remarkable compression ratio of up to 5.02x, enhances inference throughput by approximately 5x, and reduces the memory footprint by 41% compared to the FP16 full cache baseline, all while maintaining near-lossless performance.

BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference

Efficient LLM Inference with Kcache

PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

PQCache: Product Quantization-based KVCache for Long Context LLM Inference

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference

Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks

SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models

KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches

MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

SnapKV: LLM Knows What You are Looking for Before Generation

Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption

CORM: Cache Optimization with Recent Message for Large Language Model Inference

Unifying KV Cache Compression for Large Language Models with LeanKV

XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity

KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head