Scaling Laws for Linear Complexity Language Models

Xuyang Shen,Dong Li,Ruitao Leng,Zhen Qin,Weigao Sun,Yiran Zhong

2024-06-24

Abstract:The interest in linear complexity models for large language models is on the rise, although their scaling capacity remains uncertain. In this study, we present the scaling laws for linear complexity language models to establish a foundation for their scalability. Specifically, we examine the scaling behaviors of three efficient linear architectures. These include TNL, a linear attention model with data-independent decay; HGRN2, a linear RNN with data-dependent decay; and cosFormer2, a linear attention model without decay. We also include LLaMA as a baseline architecture for softmax attention for comparison. These models were trained with six variants, ranging from 70M to 7B parameters on a 300B-token corpus, and evaluated with a total of 1,376 intermediate checkpoints on various downstream tasks. These tasks include validation loss, commonsense reasoning, and information retrieval and generation. The study reveals that existing linear complexity language models exhibit similar scaling capabilities as conventional transformer-based models while also demonstrating superior linguistic proficiency and knowledge retention.

Computation and Language

What problem does this paper attempt to address?

This paper mainly discusses the scalability law of linear complexity language models. The study found that linear complexity models (such as TNL, HGRN2, and cosFormer2) can demonstrate stronger language ability and knowledge retention while maintaining similar scalability to traditional Transformer models. The paper trains models of different scales (from 70M to 7B parameters) and evaluates them on a corpus containing 300 billion tokens to determine the relationship between model performance and computational budget.

The authors propose scalability laws for linear complexity models, which describe the power-law relationship between training loss (L), model size (N), and dataset size (D) with computational budget (C). Through experiments, they found that linear complexity models require more parameters and training tokens but achieve lower loss under the same computational budget. Furthermore, these models demonstrate comparable or even better performance in downstream tasks such as language understanding, commonsense reasoning, and information retrieval and generation compared to traditional Transformer models.

The paper also analyzes the impact of model size, computational cost, and context length on model capacity, as well as the effectiveness of data dependency decay in different tasks. The results show that the aspect ratio of linear models and context length affect model capacity, while data dependency decay is beneficial in retrieval tasks.

In conclusion, this paper aims to address the scalability issue of linear complexity language models and improve their potential applications in large-scale language modeling.