Abstract:In-context learning (ICL) approaches typically leverage prompting to condition decoder-only language model generation on reference information. Just-in-time processing of a context is inefficient due to the quadratic cost of self-attention operations, and caching is desirable. However, caching transformer states can easily require almost as much space as the model parameters. When the right context isn't known in advance, caching ICL can be challenging. This work addresses these limitations by introducing models that, inspired by the encoder-decoder architecture, use cross-attention to condition generation on reference text without the prompt. More precisely, we leverage pre-trained decoder-only models and only train a small number of added layers. We use Question-Answering (QA) as a testbed to evaluate the ability of our models to perform conditional generation and observe that they outperform ICL, are comparable to fine-tuned prompted LLMs, and drastically reduce the space footprint relative to standard KV caching by two orders of magnitude.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve several key challenges in the practical application of large - language models (LLMs), especially the efficiency and performance issues related to context - conditional generation. Specifically: 1. **Inefficiency in context processing**: - When processing the context immediately, since the time complexity of the self - attention mechanism is quadratic ($O(|\text{context}|^2)$), the inference process is very inefficient. - Although caching can accelerate inference, the traditional key - value (KV) caching method requires a large amount of storage space, almost as much as the model parameters themselves. 2. **Limitations of prompt injection**: - Using prompts for context - conditional generation (such as In - Context Learning, ICL) has a high - variance problem, that is, different prompt templates will lead to significantly different results. - The prompt method is very expensive in both time and space, especially when dealing with large - scale contexts. 3. **Optimization of context caching**: - How to significantly reduce the storage space required for caching while maintaining efficient inference, thereby reducing the computational cost and communication overhead. To solve these problems, the author proposes a new architecture - XC - CACHE (Cross - Attending to Cached Context), which replaces the traditional prompt method by introducing cross - attention layers, and only caches the last - layer hidden state of the encoder instead of all intermediate states. This method not only improves the inference speed but also significantly reduces the storage space required for caching, achieving a two - order - of - magnitude reduction. ### Specific goals - **Improve inference efficiency**: By pre - processing and caching context representations, reduce the time complexity from $O(|\text{context}|^2)$ to $O(|\text{context}|(|\text{query}|+|\text{answer}|))$. - **Reduce cache space**: Compared with the traditional KV caching method, XC - CACHE can reduce the cache space requirement by more than 98%. - **Improve model performance**: In the question - answering task, the XC - CACHE model performs better than the prompt - based inference method and is close to the performance of the fine - tuned LLM. ### Method overview The author proposes a new architecture design, converting the existing decoder - only model into an encoder - decoder architecture and inserting cross - attention layers in the decoder. The specific implementation includes two variants: 1. **XC - LLAMA**: Use the frozen decoder as an encoder and directly use its last - layer representation as the context encoding. 2. **XC - LLAMA ENC**: Train a small bidirectional encoder (such as LONGFORMER) to process the context and pass its output to the decoder. Both of these methods can significantly reduce the cache space requirement while maintaining high inference performance. ### Conclusion Through these improvements, the author shows how to significantly improve the efficiency and scalability of context - conditional generation without sacrificing too much accuracy, making LLMs more practical in practical applications.

XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference

XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference

Training-Free Exponential Context Extension via Cascading KV Cache

PQCache: Product Quantization-based KVCache for Long Context LLM Inference

EPIC: Efficient Position-Independent Context Caching for Serving Large Language Models

Efficient LLM Inference with Kcache

NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time

LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference

CORM: Cache Optimization with Recent Message for Large Language Model Inference

CacheGen: KV Cache Compression and Streaming for Fast Language Model Serving

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity

XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference

Reducing Transformer Key-Value Cache Size with Cross-Layer Attention

Decoupled Context Processing for Context Augmented Language Modeling

FlashBack:Efficient Retrieval-Augmented Language Modeling for Long Context Inference

UNComp: Uncertainty-Aware Long-Context Compressor for Efficient Large Language Model Inference

VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration

Efficient Inference of Vision Instruction-Following Models with Elastic Cache

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

In-context Autoencoder for Context Compression in a Large Language Model