Abstract:Selecting a suitable pretraining dataset is crucial for both general-domain (e.g., GPT-3) and domain-specific (e.g., Codex) language models (LMs). We formalize this problem as selecting a subset of a large raw unlabeled dataset to match a desired target distribution given unlabeled target samples. Due to the scale and dimensionality of the raw text data, existing methods use simple heuristics or require human experts to manually curate data. Instead, we extend the classic importance resampling approach used in low-dimensions for LM data selection. We propose Data Selection with Importance Resampling (DSIR), an efficient and scalable framework that estimates importance weights in a reduced feature space for tractability and selects data with importance resampling according to these weights. We instantiate the DSIR framework with hashed n-gram features for efficiency, enabling the selection of 100M documents from the full Pile dataset in 4.5 hours. To measure whether hashed n-gram features preserve the aspects of the data that are relevant to the target, we define KL reduction, a data metric that measures the proximity between the selected pretraining data and the target on some feature space. Across 8 data selection methods (including expert selection), KL reduction on hashed n-gram features highly correlates with average downstream accuracy (r=0.82). When selecting data for continued pretraining on a specific domain, DSIR performs comparably to expert curation across 8 target distributions. When pretraining general-domain models (target is Wikipedia and books), DSIR improves over random selection and heuristic filtering baselines by 2-2.5% on the GLUE benchmark. Code is available at <a class="link-external link-https" href="https://github.com/p-lambda/dsir" rel="external noopener nofollow">this https URL</a>.

Data Selection Via Semi-supervised Recursive Autoencoders for SMT Domain Adaptation

Deep Joint Two-Stream Wasserstein Auto-Encoder and Selective Attention Alignment for Unsupervised Domain Adaptation

A Systematic Comparison of Data Selection Criteria for SMT Domain Adaptation

Transductive Data-Selection Algorithms for Fine-Tuning Neural Machine Translation

Generalised Unsupervised Domain Adaptation of Neural Machine Translation with Cross-Lingual Data Selection

Speech Corpora Divergence Based Unsupervised Data Selection for ASR

A Survey on Data Selection for Language Models

Domain Adaptation for Semantic Parsing.

Data Selection Strategies for Multi-Domain Sentiment Analysis

SemiEvol: Semi-supervised Fine-tuning for LLM Adaptation

Unsupervised Domain Adaptation for Referring Semantic Segmentation

Domain Adaptation for Statistical Machine Translation

Data Selection for Language Models via Importance Resampling

Adversarial Domain Adaptation For Chinese Semantic Dependency Graph Parsing

Semi-Supervised Learning for Neural Machine Translation

Semi-supervised Domain Adaptation for Semantic Segmentation

Non-Parametric Unsupervised Domain Adaptation for Neural Machine Translation

Domain Adaptation for Medical Text Translation Using Web Resources

Robust Guidance for Unsupervised Data Selection: Capturing Perplexing Named Entities for Domain-Specific Machine Translation

Data Selection via Optimal Control for Language Models

Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation.