Abstract:Selecting a suitable pretraining dataset is crucial for both general-domain (e.g., GPT-3) and domain-specific (e.g., Codex) language models (LMs). We formalize this problem as selecting a subset of a large raw unlabeled dataset to match a desired target distribution given unlabeled target samples. Due to the scale and dimensionality of the raw text data, existing methods use simple heuristics or require human experts to manually curate data. Instead, we extend the classic importance resampling approach used in low-dimensions for LM data selection. We propose Data Selection with Importance Resampling (DSIR), an efficient and scalable framework that estimates importance weights in a reduced feature space for tractability and selects data with importance resampling according to these weights. We instantiate the DSIR framework with hashed n-gram features for efficiency, enabling the selection of 100M documents from the full Pile dataset in 4.5 hours. To measure whether hashed n-gram features preserve the aspects of the data that are relevant to the target, we define KL reduction, a data metric that measures the proximity between the selected pretraining data and the target on some feature space. Across 8 data selection methods (including expert selection), KL reduction on hashed n-gram features highly correlates with average downstream accuracy (r=0.82). When selecting data for continued pretraining on a specific domain, DSIR performs comparably to expert curation across 8 target distributions. When pretraining general-domain models (target is Wikipedia and books), DSIR improves over random selection and heuristic filtering baselines by 2-2.5% on the GLUE benchmark. Code is available at <a class="link-external link-https" href="https://github.com/p-lambda/dsir" rel="external noopener nofollow">this https URL</a>.

DsDm: Model-Aware Dataset Selection with Datamodels

Data Debiasing with Datamodels (D3M): Improving Subgroup Robustness via Data Selection

Towards Accelerated Model Training via Bayesian Data Selection

Data Selection for Language Models via Importance Resampling

Model Selection Techniques -- An Overview

Accelerating Dataset Distillation Via Model Augmentation

A Survey on Data Selection for Language Models

TSDS: Data Selection for Task-Specific Model Finetuning

An Efficient Source Model Selection Framework in Model Databases

Optimal subdata selection for linear model selection

Automated Model Selection for Tabular Data

SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilation

Exploring Dataset-Scale Indicators of Data Quality

A CLIP-Powered Framework for Robust and Generalizable Data Selection

Designing Data: Proactive Data Collection and Iteration for Machine Learning

A Fast Automated Model Selection Approach Based on Collaborative Knowledge

A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data

Reinforced Data Sampling for Model Diversification

A Case Study on a Sustainable Framework for Ethically Aware Predictive Modeling

Dantzig Selector with an Approximately Optimal Denoising Matrix and its Application to Reinforcement Learning

Redeeming Data Science by Decision Modelling