Abstract:In-context learning (ICL) facilitates large language models (LLMs) exhibiting spectacular emergent capabilities in various scenarios. Unfortunately, introducing demonstrations easily makes the prompt length explode, bringing a significant burden to hardware. In addition, random demonstrations usually achieve limited improvements in ICL, necessitating demonstration selection among accessible candidates. Previous studies introduce extra modules to perform demonstration compression or selection independently. In this paper, we propose an ICL framework UniICL, which Unifies demonstration selection and compression, and final response generation via a single frozen LLM. Specifically, UniICL first projects actual demonstrations and inference text inputs into short virtual tokens, respectively. Then, virtual tokens are applied to select suitable demonstrations by measuring semantic similarity within latent space among candidate demonstrations and inference input. Finally, inference text inputs together with selected virtual demonstrations are fed into the same frozen LLM for response generation. Notably, UniICL is a parameter-efficient framework that only contains 17M trainable parameters originating from the projection layer. We conduct experiments and analysis over in- and out-domain datasets of both generative and understanding tasks, encompassing ICL scenarios with plentiful and limited demonstration candidates. Results show that UniICL effectively unifies $12 \times$ compression, demonstration selection, and response generation, efficiently scaling up the baseline from 4-shot to 64-shot ICL in IMDb with 24 GB CUDA allocation

Effective In-Context Example Selection through Data Compression

In-Context Compositional Generalization for Large Vision-Language Models

Finding Support Examples for In-Context Learning

In-Context Demonstration Selection with Cross Entropy Difference

Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability

In-Context Former: Lightning-fast Compressing Context for Large Language Model

Context Compression for Auto-regressive Transformers with Sentinel Tokens

Adapting Language Models to Compress Contexts

Evaluating Large Language Models for Generalization and Robustness via Data Compression

Context Compression and Extraction: Efficiency Inference of Large Language Models

Going Beyond Word Matching: Syntax Improves In-context Example Selection for Machine Translation

Extending Context Window of Large Language Models via Semantic Compression

Perception Compressor:A training-free prompt compression method in long context scenarios

Recurrent Context Compression: Efficiently Expanding the Context Window of LLM

Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression

How Do In-Context Examples Affect Compositional Generalization?

LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models

Unifying Demonstration Selection and Compression for In-Context Learning

Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selection

Compositional Exemplars for In-context Learning

Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning