Abstract:Spurred by advancements in scale, large language models (LLMs) have demonstrated strong few-shot learning ability via in-context learning (ICL). However, the performance of ICL has been shown to be highly sensitive to the selection of few-shot demonstrations. Selecting the most suitable examples as context remains an ongoing challenge and an open problem. Existing literature has highlighted the importance of selecting examples that are diverse or semantically similar to the test sample while ignoring the fact that the optimal selection dimension, i.e., diversity or similarity, is task-specific. Based on how the test sample is answered, we propose Iterative Demonstration Selection (IDS) to leverage the merits of both dimensions. Using zero-shot chain-of-thought reasoning (Zero-shot-CoT), IDS iteratively selects examples that are diverse but still strongly correlated with the test sample as ICL demonstrations. Specifically, IDS applies Zero-shot-CoT to the test sample before demonstration selection. The output reasoning path is then used to choose demonstrations that are prepended to the test sample for inference. The generated answer is followed by its corresponding reasoning path for extracting a new set of demonstrations in the next iteration. After several iterations, IDS adopts majority voting to obtain the final result. Through extensive experiments on tasks including reasoning, question answering, and topic classification, we demonstrate that IDS can consistently outperform existing ICL demonstration selection methods.

Iterative Forward Tuning Boosts In-Context Learning in Language Models

Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning

In-Context Learning Demonstration Selection via Influence Analysis

Revisiting Demonstration Selection Strategies in In-Context Learning

Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning

In-Context Learning with Iterative Demonstration Selection

Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning

Large Language Models Know What Makes Exemplary Contexts

Exploring the Relationship between In-Context Learning and Instruction Tuning

What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning

Enhancing In-Context Learning via Implicit Demonstration Augmentation

Misconfidence-based Demonstration Selection for LLM In-Context Learning

Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning

Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning

Hint-enhanced In-Context Learning wakes Large Language Models up for knowledge-intensive tasks

In-context Learning with Retrieved Demonstrations for Language Models: A Survey

Focused Large Language Models are Stable Many-Shot Learners

Dynamic Demonstrations Controller for In-Context Learning

ICLEval: Evaluating In-Context Learning Ability of Large Language Models

Learning to Retrieve Iteratively for In-Context Learning

In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting