Abstract:A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation. However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference. Our approach is based on the observation that KV cache states exhibit high similarity between the adjacent layers in the middle-to-deep portion of LLMs. To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead. Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity. We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with 4-bit MiniCache achieves a remarkable compression ratio of up to 5.02x, enhances inference throughput by approximately 5x, and reduces the memory footprint by 41% compared to the FP16 full cache baseline, all while maintaining near-lossless performance.

LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching

An LLM-Tool Compiler for Fused Parallel Function Calling

Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching

Deliberation in Latent Space via Differentiable Cache Augmentation

SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models

On Optimal Caching and Model Multiplexing for Large Model Inference

DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving

Efficient LLM Inference with Kcache

GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching

CORM: Cache Optimization with Recent Message for Large Language Model Inference

Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention

MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

CaM: Cache Merging for Memory-efficient LLMs Inference

NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time

Edge-LLM: A Collaborative Framework for Large Language Model Serving in Edge Computing

Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption

AcceLLM: Accelerating LLM Inference using Redundancy for Load Balancing and Data Locality

Large Language Models as Tool Makers

InstCache: A Predictive Cache for LLM Serving

MeanCache: User-Centric Semantic Cache for Large Language Model Based Web Services