Abstract:The ability of large language models (LLMs) to $``$learn in context$"$ based on the provided prompt has led to an explosive growth in their use, culminating in the proliferation of AI assistants such as ChatGPT, Claude, and Bard. These AI assistants are known to be robust to minor prompt modifications, mostly due to alignment techniques that use human feedback. In contrast, the underlying pre-trained LLMs they use as a backbone are known to be brittle in this respect. Building high-quality backbone models remains a core challenge, and a common approach to assessing their quality is to conduct few-shot evaluation. Such evaluation is notorious for being highly sensitive to minor prompt modifications, as well as the choice of specific in-context examples. Prior work has examined how modifying different elements of the prompt can affect model performance. However, these earlier studies tended to concentrate on a limited number of specific prompt attributes and often produced contradictory results. Additionally, previous research either focused on models with fewer than 15 billion parameters or exclusively examined black-box models like GPT-3 or PaLM, making replication challenging. In the present study, we decompose the entire prompt into four components: task description, demonstration inputs, labels, and inline instructions provided for each demonstration. We investigate the effects of structural and semantic corruptions of these elements on model performance. We study models ranging from 1.5B to 70B in size, using ten datasets covering classification and generation tasks. We find that repeating text within the prompt boosts model performance, and bigger models ($\geq$30B) are more sensitive to the semantics of the prompt. Finally, we observe that adding task and inline instructions to the demonstrations enhances model performance even when the instructions are semantically corrupted.

Structured Prompting: Scaling In-Context Learning to 1,000 Examples

Deconstructing In-Context Learning: Understanding Prompts via Corruption

Fairness-guided Few-shot Prompting for Large Language Models

Efficient Prompting via Dynamic In-Context Learning

Large Language Models Prompting With Episodic Memory

Learning How To Ask: Cycle-Consistency Refines Prompts in Multimodal Foundation Models

Scaling In-Context Demonstrations with Structured Attention

Enhancing In-Context Learning via Implicit Demonstration Augmentation

Prompt Space Optimizing Few-shot Reasoning Success with Large Language Models

Helping Language Models Learn More: Multi-dimensional Task Prompt for Few-shot Tuning

Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies

Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning

Boosted Prompt Ensembles for Large Language Models

S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning

Structured Prompt Tuning

CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learning

Learning To Retrieve Prompts for In-Context Learning

PhaseEvo: Towards Unified In-Context Prompt Optimization for Large Language Models

Prompting Language Models for Linguistic Structure

Large Language Model Prompt Chaining for Long Legal Document Classification