LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression

Zhuoshi Pan,Qianhui Wu,Huiqiang Jiang,Menglin Xia,Xufang Luo,Jue Zhang,Qingwei Lin,Victor Rühle,Yuqing Yang,Chin-Yew Lin,H. Vicky Zhao,Lili Qiu,Dongmei Zhang

2024-08-12

Abstract:This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. Considering the redundancy in natural language, existing approaches compress prompts by removing tokens or lexical units according to their information entropy obtained from a causal language model such as LLaMa-7B. The challenge is that information entropy may be a suboptimal compression metric: (i) it only leverages unidirectional context and may fail to capture all essential information needed for prompt compression; (ii) it is not aligned with the prompt compression objective. To address these issues, we propose a data distillation procedure to derive knowledge from an LLM to compress prompts without losing crucial information, and meantime, introduce an extractive text compression dataset. We formulate prompt compression as a token classification problem to guarantee the faithfulness of the compressed prompt to the original one, and use a Transformer encoder as the base architecture to capture all essential information for prompt compression from the full bidirectional context. Our approach leads to lower latency by explicitly learning the compression objective with smaller models such as XLM-RoBERTa-large and mBERT. We evaluate our method on both in-domain and out-of-domain datasets, including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its small size, our model shows significant performance gains over strong baselines and demonstrates robust generalization ability across different LLMs. Additionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.6x-2.9x with compression ratios of 2x-5x. Our code is available at <a class="link-external link-https" href="https://aka.ms/LLMLingua-2" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Machine Learning

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is **information loss and efficiency issues in task - agnostic prompt compression**. Specifically, existing prompt - compression methods face the following challenges:

1. **Limitations of information entropy as a compression metric**: Existing methods usually use the information entropy calculated by causal language models (such as LLaMA - 7B) to compress prompts, but information entropy only utilizes one - way context and may not be able to capture all necessary information and is not fully aligned with the goals of prompt compression.

2. **Importance of bidirectional context**: In order to perform prompt compression better, it is necessary to make full use of all necessary information in the bidirectional context.

To solve these problems, the author proposes a method based on data distillation to extract knowledge from large - language models (LLMs) to compress prompts, and at the same time introduces a new extractive - text - compression dataset. By modeling prompt compression as a binary - classification problem (keep or discard) and using a Transformer encoder as a feature extractor, it is ensured that the compressed prompts significantly improve the compression efficiency while maintaining the fidelity of the original content.

### Main contributions
1. **Data distillation process**: Extract knowledge from GPT - 4 to generate compressed prompts without losing key information, and construct an extractive - text - compression dataset containing the original text and its compressed version.
2. **Prompt compression as a classification task**: Model prompt compression as a binary - classification problem, and use the predicted probability as a compression metric, thereby ensuring the fidelity of the compressed prompts.
3. **Experimental verification**: Extensive experiments were carried out on multiple datasets, including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Although the model is smaller, its performance is significantly better than existing methods, and the end - to - end latency is accelerated.

Through these improvements, this paper demonstrates the robustness and generalization ability of its method on different tasks and models, while significantly improving the compression speed and efficiency.