Abstract:Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web. In this work, we propose Web Rephrase Augmented Pre-training ($\textbf{WRAP}$) that uses an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as "like Wikipedia" or in "question-answer format" to jointly pre-train LLMs on real and synthetic rephrases. First, we show that using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training by $\sim3x$. At the same pre-training compute budget, it improves perplexity by more than 10% on average across different subsets of the Pile, and improves zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we investigate the impact of the re-phrasing style on the performance of the model, offering insights into how the composition of the training data can impact the performance of LLMs in OOD settings. Our gains are attributed to the fact that re-phrased synthetic data has higher utility than just real data because it (i) incorporates style diversity that closely reflects downstream evaluation style, and (ii) has higher 'quality' than web-scraped data.

Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training

German Text Simplification: Finetuning Large Language Models with Semi-Synthetic Data

Data and Approaches for German Text simplification -- towards an Accessibility-enhanced Communication

DEPLAIN: A German Parallel Corpus with Intralingual Translations into Plain Language for Sentence and Document Simplification

Pre-Trained Acoustic-and-Textual Modeling for End-To-End Speech-To-Text Translation.

Exploring Automatic Text Simplification of German Narrative Documents

Mitigating Data Scarcity for Large Language Models

Investigating Text Simplification Evaluation

Comprehensive Study on German Language Models for Clinical and Biomedical Text Understanding

Historical German Text Normalization Using Type- and Token-Based Language Modeling

Semi-Supervised Text Simplification with Back-Translation and Asymmetric Denoising Autoencoders

LLäMmlein: Compact and Competitive German-Only Language Models from Scratch

Improving data augmentation for low resource speech-to-text translation with diverse paraphrasing

Exploiting Biased Models to De-bias Text: A Gender-Fair Rewriting Model

EASSE-DE: Easier Automatic Sentence Simplification Evaluation for German

Investigating Neural Machine Translation for Low-Resource Languages: Using Bavarian as a Case Study

Efficient Speech Translation with Pre-trained Models

Pre-Training a Language Model Without Human Language

Enhancing Pre-trained Language Model with Lexical Simplification

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

Scalable Data Ablation Approximations for Language Models through Modular Training and Merging