Abstract:We propose expanding the shared Transformer module to produce and initialize Transformers of varying depths, enabling adaptation to diverse resource constraints. Drawing an analogy to genetic expansibility, we term such module as learngene. To identify the expansion mechanism, we delve into the relationship between the layer's position and its corresponding weight value, and find that linear function appropriately approximates this relationship. Building on this insight, we present Transformer as Linear Expansion of learnGene (TLEG), a novel approach for flexibly producing and initializing Transformers of diverse depths. Specifically, to learn learngene, we firstly construct an auxiliary Transformer linearly expanded from learngene, after which we train it through employing soft distillation. Subsequently, we can produce and initialize Transformers of varying depths via linearly expanding the well-trained learngene, thereby supporting diverse downstream scenarios. Extensive experiments on ImageNet-1K demonstrate that TLEG achieves comparable or better performance in contrast to many individual models trained from scratch, while reducing around 2x training cost. When transferring to several downstream classification datasets, TLEG surpasses existing initialization methods by a large margin (e.g., +6.87% on iNat 2019 and +7.66% on CIFAR-100). Under the situation where we need to produce models of varying depths adapting for different resource constraints, TLEG achieves comparable results while reducing around 19x parameters stored to initialize these models and around 5x pre-training costs, in contrast to the pre-training and fine-tuning approach. When transferring a fixed set of parameters to initialize different models, TLEG presents better flexibility and competitive performance while reducing around 2.9x parameters stored to initialize, compared to the pre-training approach.

Transforlearn: Interactive visual tutorial for the transformer model

TransforLearn: Interactive Visual Tutorial for the Transformer Model.

Transformer Explainer: Interactive Learning of Text-Generative Models

An Introduction to Transformers

Learning Transformer Programs

Understanding the Difficulty of Training Transformers

How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations

Introduction to Transformers: an NLP Perspective

Transformers Meet Visual Learning Understanding: A Comprehensive Review

Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions

What Can Transformers Learn In-Context? A Case Study of Simple Function Classes

Breaking through the learning plateaus of in-context learning in Transformer

LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models

Dodrio: Exploring Transformer Models with Interactive Visualization

Transformers in computational visual media: A survey

How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding

Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View

T3-Vis - Visual Analytic for Training and Fine-Tuning Transformers in NLP.

Transformer As Linear Expansion of Learngene

Understanding and Improving Transformer from a Multi-Particle Dynamic System Point of View.

Emergence of Abstractions: Concept Encoding and Decoding Mechanism for In-Context Learning in Transformers