Abstract:A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation. However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference. Our approach is based on the observation that KV cache states exhibit high similarity between the adjacent layers in the middle-to-deep portion of LLMs. To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead. Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity. We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with 4-bit MiniCache achieves a remarkable compression ratio of up to 5.02x, enhances inference throughput by approximately 5x, and reduces the memory footprint by 41% compared to the FP16 full cache baseline, all while maintaining near-lossless performance.

Palu: Compressing KV-Cache with Low-Rank Projection

ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification

MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection

PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy

Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression

CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

Lossless KV Cache Compression to 2%

Effectively Compress KV Heads for LLM

No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization

Unifying KV Cache Compression for Large Language Models with LeanKV

KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head

AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantization

PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

QAQ: Quality Adaptive Quantization for LLM KV Cache

EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importance

PQCache: Product Quantization-based KVCache for Long Context LLM Inference