Abstract:Long prompt leads to huge hardware costs when using transformer-based Large Language Models (LLMs). Unfortunately, many tasks, such as summarization, inevitably introduce long documents, and the wide application of in-context learning easily makes the prompt length explode. This paper proposes a Self-Compressor (SelfCP), which employs the target LLM itself to compress over-limit prompts into dense vectors while keeping the allowed prompts unmodified. Dense vectors are then projected into dense tokens via a learnable connector to make the same LLM unburden to understand. The connector is supervised-tuned under the language modeling objective of the LLM on relatively long texts selected from publicly accessed datasets, involving an instruction dataset to make SelfCP respond to various prompts, while the target LLM keeps frozen during training. We build the lightweight SelfCP upon 2 different backbones with merely 17M learnable parameters originating from the connector and a learnable embedding. Evaluation on both English and Chinese benchmarks demonstrate that SelfCP effectively substitutes 12$\times$ over-limit prompts with dense tokens to reduce memory costs and booster inference throughputs, yet improving response quality. The outstanding performance brings an efficient solution for LLMs to tackle long prompts without training LLMs from scratch.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: when using Transformer - based large - scale language models (LLMs), the huge hardware cost and efficiency problems caused by long prompts. Specifically: 1. **Memory explosion brought by long prompts**: - When using large - scale language models, long prompts will lead to excessive memory consumption because the performance of these models depends on carefully designed prompts, and long prompts will exceed the context window limits of the models. 2. **Long prompts caused by task requirements**: - Many natural language processing tasks (such as summary generation, question - answering systems, etc.) need to include long documents as input, which inevitably introduces long prompts. 3. **Application of in - context learning**: - The wide application of in - context learning (ICL) further increases the length of prompts because ICL usually needs to introduce multiple examples. To solve these problems, the author proposes a method named SelfCP (Self - Compressor), which uses the frozen large - scale language model itself to compress overly long prompts. In this way, SelfCP can compress overly long prompts into dense vectors, thereby reducing memory usage and improving inference efficiency while maintaining generation quality. Specifically, the main contributions of SelfCP include: - **Using the frozen large - scale language model itself for the first time** to compress overly long prompts. The compressed prompts only account for 1/12 of the memory tokens of the original prompts, being more general and less costly. - **Proposing a more dialectical perspective on prompt compression**, achieving a balance among training cost, inference efficiency and generation quality. Through these improvements, SelfCP can effectively handle long prompts without retraining large - scale language models, thereby reducing hardware costs and improving processing efficiency.

SelfCP: Compressing Over-Limit Prompt via the Frozen Large Language Model Itself

500xCompressor: Generalized Prompt Compression for Large Language Models

Perception Compressor:A training-free prompt compression method in long context scenarios

LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models

Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt

Prompt Compression for Large Language Models: A Survey

LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression

Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability

Adapting LLMs for Efficient Context Processing through Soft Prompt Compression

Learning to Compress Prompt in Natural Language Formats

From Reading to Compressing: Exploring the Multi-document Reader for Prompt Compression

Discrete Prompt Compression With Reinforcement Learning

Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models

Parse Trees Guided LLM Prompt Compression

Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference

Extending Context Window of Large Language Models via Semantic Compression

LanguaShrink: Reducing Token Overhead with Psycholinguistics

Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles

Network-aided Efficient Large Language Model Services With Denoising-inspired Prompt Compression

Adapting Language Models to Compress Contexts