Abstract:Large Language Models (LLMs) have emerged as a milestone in artificial intelligence, and their performance can improve as the model size increases. However, this scaling brings great challenges to training and inference efficiency, particularly for deploying LLMs in resource-constrained environments, and the scaling trend is becoming increasingly unsustainable. This paper introduces the concept of ``\textit{capacity density}'' as a new metric to evaluate the quality of the LLMs across different scales and describes the trend of LLMs in terms of both effectiveness and efficiency. To calculate the capacity density of a given target LLM, we first introduce a set of reference models and develop a scaling law to predict the downstream performance of these reference models based on their parameter sizes. We then define the \textit{effective parameter size} of the target LLM as the parameter size required by a reference model to achieve equivalent performance, and formalize the capacity density as the ratio of the effective parameter size to the actual parameter size of the target LLM. Capacity density provides a unified framework for assessing both model effectiveness and efficiency. Our further analysis of recent open-source base LLMs reveals an empirical law (the densing law)that the capacity density of LLMs grows exponentially over time. More specifically, using some widely used benchmarks for evaluation, the capacity density of LLMs doubles approximately every three months. The law provides new perspectives to guide future LLM development, emphasizing the importance of improving capacity density to achieve optimal results with minimal computational overhead.

Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

Physics of Language Models: Part 3.2, Knowledge Manipulation

Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

Can Language Models Act as Knowledge Bases at Scale?

Observational Scaling Laws and the Predictability of Language Model Performance

Scaling Laws for Linear Complexity Language Models

Scaling Laws for Fact Memorization of Large Language Models

Densing Law of LLMs

Large Language Models and Knowledge Graphs: Opportunities and Challenges

An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models

Scaling Laws for Neural Language Models

Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process

Revealing the structure of language model capabilities

Quantifying the Capabilities of LLMs across Scale and Precision

The Quantization Model of Neural Scaling

Evaluating Computational Language Models with Scaling Properties of Natural Language

Measuring and Modifying Factual Knowledge in Large Language Models

Scaling laws for language encoding models in fMRI

Defining Knowledge: Bridging Epistemology and Large Language Models

Augmented Large Language Models with Parametric Knowledge Guiding

Scaling Law for Language Models Training Considering Batch Size