Abstract:Many recent approaches to natural language tasks are built on the remarkable abilities of large language models. Large language models can perform in-context learning, where they learn a new task from a few task demonstrations, without any parameter updates. This work examines the implications of in-context learning for the creation of datasets for new natural language tasks. Departing from recent in-context learning methods, we formulate an annotation-efficient, two-step framework: selective annotation that chooses a pool of examples to annotate from unlabeled data in advance, followed by prompt retrieval that retrieves task examples from the annotated pool at test time. Based on this framework, we propose an unsupervised, graph-based selective annotation method, voke-k, to select diverse, representative examples to annotate. Extensive experiments on 10 datasets (covering classification, commonsense reasoning, dialogue, and text/code generation) demonstrate that our selective annotation method improves the task performance by a large margin. On average, vote-k achieves a 12.9%/11.4% relative gain under an annotation budget of 18/100, as compared to randomly selecting examples to annotate. Compared to state-of-the-art supervised finetuning approaches, it yields similar performance with 10-100x less annotation cost across 10 tasks. We further analyze the effectiveness of our framework in various scenarios: language models with varying sizes, alternative selective annotation methods, and cases where there is a test data domain shift. We hope that our studies will serve as a basis for data annotations as large language models are increasingly applied to new tasks. Our code is available at https://github.com/HKUNLP/icl-selective-annotation.

From Random to Informed Data Selection: A Diversity-Based Approach to Optimize Human Annotation and Few-Shot Learning

Crowdsourcing Detection of Sampling Biases in Image Datasets

When Low Resource NLP Meets Unsupervised Language Model: Meta-Pretraining then Meta-Learning for Few-Shot Text Classification (Student Abstract)

Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement

Navigating Towards Fairness with Data Selection

Diversity Helps: Unsupervised Few-shot Learning Via Distribution Shift-based Data Augmentation

On Training Instance Selection for Few-Shot Neural Text Generation

Selective Annotation Makes Language Models Better Few-Shot Learners

Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions

Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Models

Rethinking Data Selection at Scale: Random Selection is Almost All You Need

Use Random Selection for Now: Investigation of Few-Shot Selection Strategies in LLM-based Text Augmentation for Classification

Automatic Combination of Sample Selection Strategies for Few-Shot Learning

SelectNAdapt: Support Set Selection for Few-Shot Domain Adaptation

Improving Data Efficiency via Curating LLM-Driven Rating Systems

Diversity-Infused Network for Unsupervised Few-Shot Remote Sensing Scene Classification

Detecting discriminatory risk through data annotation based on Bayesian inferences

Focus Nuance and Toward Diversity: Exploring Domain-Specific Fine-Grained Few-Shot Recognition.

Autonomous Perception and Adaptive Standardization for Few-Shot Learning

Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion

Leveraging Biases in Large Language Models: "bias-kNN'' for Effective Few-Shot Learning