Abstract:Large Language Models (LLMs) have made remarkable progress in processing extensive contexts, with the Key-Value (KV) cache playing a vital role in enhancing their performance. However, the growth of the KV cache in response to increasing input length poses challenges to memory and time efficiency. To address this problem, this paper introduces SnapKV, an innovative and fine-tuning-free approach that efficiently minimizes KV cache size while still delivering comparable performance in real-world applications. We discover that each attention head in the model consistently focuses on specific prompt attention features during generation. Meanwhile, this robust pattern can be obtained from an 'observation' window located at the end of the prompts. Drawing on this insight, SnapKV automatically compresses KV caches by selecting clustered important KV positions for each attention head. Our approach significantly reduces the growing computational overhead and memory footprint when processing long input sequences. Specifically, SnapKV achieves a consistent decoding speed with a 3.6x increase in generation speed and an 8.2x enhancement in memory efficiency compared to the baseline when processing inputs of 16K tokens. At the same time, it maintains comparable performance to the baseline models across 16 long sequence datasets. Moreover, SnapKV can process up to 380K context tokens on a single A100-80GB GPU using HuggingFace implementation with minor changes, exhibiting only a negligible accuracy drop in the Needle-in-a-Haystack test. Further comprehensive studies suggest SnapKV's potential for practical applications.

Leveraging Speculative Sampling and KV-Cache Optimizations Together for Generative AI using OpenVINO

Optimized Speculative Sampling for GPU Hardware Accelerators

BASS: Batched Attention-optimized Speculative Sampling

InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

Fine-Tuning Generative Models as an Inference Method for Robotic Tasks

GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM

Accelerating Large Language Model Decoding with Speculative Sampling

Sample as You Infer: Predictive Coding With Langevin Dynamics

KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation

The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation

Kernel Looping: Eliminating Synchronization Boundaries for Peak Inference Performance

Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference

Inference Optimization of Foundation Models on AI Accelerators

Sampling and Ranking for Digital Ink Generation on a tight computational budget

Adaptive Sampling for Deep Learning via Efficient Nonparametric Proxies

Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs

Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models

Importance Sampled Stochastic Optimization for Variational Inference

PaSS: Parallel Speculative Sampling

SnapKV: LLM Knows What You are Looking for Before Generation