The Impact of Depth on Compositional Generalization in Transformer Language Models

Jackson Petty,Sjoerd van Steenkiste,Ishita Dasgupta,Fei Sha,Dan Garrette,Tal Linzen

2024-04-11

Abstract:To process novel sentences, language models (LMs) must generalize compositionally -- combine familiar elements in new ways. What aspects of a model's structure promote compositional generalization? Focusing on transformers, we test the hypothesis, motivated by theoretical and empirical work, that deeper transformers generalize more compositionally. Simply adding layers increases the total number of parameters; to address this confound between depth and size, we construct three classes of models which trade off depth for width such that the total number of parameters is kept constant (41M, 134M and 374M parameters). We pretrain all models as LMs and fine-tune them on tasks that test for compositional generalization. We report three main conclusions: (1) after fine-tuning, deeper models generalize more compositionally than shallower models do, but the benefit of additional layers diminishes rapidly; (2) within each family, deeper models show better language modeling performance, but returns are similarly diminishing; (3) the benefits of depth for compositional generalization cannot be attributed solely to better performance on language modeling. Because model latency is approximately linear in the number of layers, these results lead us to the recommendation that, with a given total parameter budget, transformers can be made shallower than is typical without sacrificing performance.

Computation and Language

What problem does this paper attempt to address?

This paper explores the impact of depth on the compositional generalization capability of Transformer language models. The study hypothesizes that increasing the depth of the model can improve its performance on tasks that require compositional generalization. To address the confusion between depth and size (number of parameters), the authors constructed models of different depths while keeping the total number of parameters constant. The experiments show that after fine-tuning, deeper models perform better than shallow models on compositional generalization, but the benefits of additional layers quickly diminish. Furthermore, depth is positively correlated with language modeling performance, but with diminishing returns. The study also finds that the benefits of depth on compositional generalization cannot solely be attributed to improved language modeling performance. Considering the issue of latency, it is suggested that, given a given parameter budget, Transformer models that are shallower than typical cases can be built without sacrificing performance.

The Impact of Depth on Compositional Generalization in Transformer Language Models

An Intrinsic Dimension Perspective of Transformers for Sequential Modeling

Explicitly Modeling Adaptive Depths for Transformer

Deep Transformers with Latent Depth

Faith and Fate: Limits of Transformers on Compositionality

Faster Depth-Adaptive Transformers

Are More Layers Beneficial to Graph Transformers?

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers

Theoretical limitations of multi-layer Transformer

Syntax-Guided Transformers: Elevating Compositional Generalization and Grounding in Multimodal Environments

What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks

Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically

Emergence of a High-Dimensional Abstraction Phase in Language Transformers

Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data

Exploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performance

Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models

Multi-Path Transformer is Better: A Case Study on Neural Machine Translation.

Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules

Out-of-distribution generalization via composition: a lens through induction heads in Transformers

Towards Understanding the Relationship between In-context Learning and Compositional Generalization

Transformer Layers as Painters