Ian R. McKenzie,Alexander Lyzhov,Michael Pieler,Alicia Parrish,Aaron Mueller,Ameya Prabhu,Euan McLean,Aaron Kirtland,Alexis Ross,Alisa Liu,Andrew Gritsevskiy,Daniel Wurgaft,Derik Kauffman,Gabriel Recchia,Jiacheng Liu,Joe Cavanagh,Max Weiss,Sicong Huang,Floating Droid,Tom Tseng,Tomasz Korbak,Xudong Shen,Yuhui Zhang,Zhengping Zhou,Najoung Kim,Samuel R. Bowman,Ethan Perez

Abstract:Work on scaling laws has found that large language models (LMs) show predictable improvements to overall loss with increased scale (model size, training data, and compute). Here, we present evidence for the claim that LMs may show inverse scaling, or worse task performance with increased scale, e.g., due to flaws in the training objective and data. We present empirical evidence of inverse scaling on 11 datasets collected by running a public contest, the Inverse Scaling Prize, with a substantial prize pool. Through analysis of the datasets, along with other examples found in the literature, we identify four potential causes of inverse scaling: (i) preference to repeat memorized sequences over following in-context instructions, (ii) imitation of undesirable patterns in the training data, (iii) tasks containing an easy distractor task which LMs could focus on, rather than the harder real task, and (iv) correct but misleading few-shot demonstrations of the task. We release the winning datasets at <a class="link-external link-https" href="https://inversescaling.com/data" rel="external noopener nofollow">this https URL</a> to allow for further investigation of inverse scaling. Our tasks have helped drive the discovery of U-shaped and inverted-U scaling trends, where an initial trend reverses, suggesting that scaling trends are less reliable at predicting the behavior of larger-scale models than previously understood. Overall, our results suggest that there are tasks for which increased model scale alone may not lead to progress, and that more careful thought needs to go into the data and objectives for training language models.

Scaling Laws for Discriminative Speech Recognition Rescoring Models

Discriminative Speech Recognition Rescoring with Pre-trained Language Models

Distillation Strategies for Discriminative Speech Recognition Rescoring

Speech Recognition Rescoring with Large Speech-Text Foundation Models

Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition

Scaling Laws for Neural Language Models

Scaling Laws for Multilingual Language Models

Scaling Laws For Dense Retrieval

Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments

Scaling Laws for Transfer

Language models scale reliably with over-training and on downstream tasks

Inverse Scaling: When Bigger Isn't Better

A Hitchhiker's Guide to Scaling Law Estimation

Scaling Laws for Precision

Observational Scaling Laws and the Predictability of Language Model Performance

Beyond Oversmoothing: Evaluating DDPM and MSE for Scalable Speech Synthesis in ASR

An Empirical Study of Scaling Law for OCR

Scaling laws for language encoding models in fMRI

Scaling Laws for Downstream Task Performance of Large Language Models

Scaling Up Deliberation for Multilingual ASR

Scaling Properties of Speech Language Models