Abstract:A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation. However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference. Our approach is based on the observation that KV cache states exhibit high similarity between the adjacent layers in the middle-to-deep portion of LLMs. To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead. Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity. We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with 4-bit MiniCache achieves a remarkable compression ratio of up to 5.02x, enhances inference throughput by approximately 5x, and reduces the memory footprint by 41% compared to the FP16 full cache baseline, all while maintaining near-lossless performance.

MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache

MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference

DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity

KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

SnapKV: LLM Knows What You are Looking for Before Generation

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

PQCache: Product Quantization-based KVCache for Long Context LLM Inference

ThinK: Thinner Key Cache by Query-Driven Pruning

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression

Efficient LLM Inference with Kcache

CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios

PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression

SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction

LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy

Lossless KV Cache Compression to 2%