Abstract:In-context learning (ICL) facilitates large language models (LLMs) exhibiting spectacular emergent capabilities in various scenarios. Unfortunately, introducing demonstrations easily makes the prompt length explode, bringing a significant burden to hardware. In addition, random demonstrations usually achieve limited improvements in ICL, necessitating demonstration selection among accessible candidates. Previous studies introduce extra modules to perform demonstration compression or selection independently. In this paper, we propose an ICL framework UniICL, which Unifies demonstration selection and compression, and final response generation via a single frozen LLM. Specifically, UniICL first projects actual demonstrations and inference text inputs into short virtual tokens, respectively. Then, virtual tokens are applied to select suitable demonstrations by measuring semantic similarity within latent space among candidate demonstrations and inference input. Finally, inference text inputs together with selected virtual demonstrations are fed into the same frozen LLM for response generation. Notably, UniICL is a parameter-efficient framework that only contains 17M trainable parameters originating from the projection layer. We conduct experiments and analysis over in- and out-domain datasets of both generative and understanding tasks, encompassing ICL scenarios with plentiful and limited demonstration candidates. Results show that UniICL effectively unifies $12 \times$ compression, demonstration selection, and response generation, efficiently scaling up the baseline from 4-shot to 64-shot ICL in IMDb with 24 GB CUDA allocation

In-Context Learning Demonstration Selection via Influence Analysis

Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning

Revisiting Demonstration Selection Strategies in In-Context Learning

In-Context Compositional Generalization for Large Vision-Language Models

Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning

In-Context Learning with Iterative Demonstration Selection

Unraveling Indirect In-Context Learning Using Influence Functions

Misconfidence-based Demonstration Selection for LLM In-Context Learning

Curriculum Demonstration Selection for In-Context Learning

Comparable Demonstrations Are Important in In-Context Learning: A Novel Perspective on Demonstration Selection

Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process

Enhancing In-Context Learning via Implicit Demonstration Augmentation

In-context Learning with Retrieved Demonstrations for Language Models: A Survey

Unifying Demonstration Selection and Compression for In-Context Learning

ParaICL: Towards Robust Parallel In-Context Learning

Demonstration Augmentation for Zero-shot In-context Learning

In-context Example Selection with Influences

Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning

Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learning

Dynamic Demonstrations Controller for In-Context Learning

Iterative Forward Tuning Boosts In-Context Learning in Language Models