Abstract:The fast growing capabilities of large-scale deep learning models, such as Bert, GPT and ViT, are revolutionizing the landscape of NLP, CV and many other domains. Training such models, however, poses an unprecedented demand for computing power, which incurs exponentially increasing energy cost and carbon dioxide emissions. It is thus critical to develop efficient training solutions to reduce the training costs. Motivated by a set of key observations of inter- and intra-layer similarities among feature maps and attentions that can be identified from typical training processes, we propose a multi-level framework for training acceleration. Specifically, the framework is based on three basic operators, Coalescing, De-coalescing and Interpolation, which can be orchestrated to build a multi-level training framework. The framework consists of a V-cycle training process, which progressively down- and up-scales the model size and projects the parameters between adjacent levels of models via coalescing and de-coalescing. The key idea is that a smaller model that can be trained for fast convergence and the trained parameters provides high-qualities intermediate solutions for the next level larger network. The interpolation operator is designed to break the symmetry of neurons incurred by de-coalescing for better convergence performance. Our experiments on transformer-based language models (e.g. Bert, GPT) as well as a vision model (e.g. DeiT) prove that the proposed framework reduces the computational cost by about 20% on training BERT/GPT-Base models and up to 51.6% on training the BERT-Large model while preserving the performance.

Hierarchical Transformers Are More Efficient Language Models

Language Models as Hierarchy Encoders

Hierarchical Transformer for Scalable Graph Learning

Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling

Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding

Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models

How transformers learn structured data: insights from hierarchical filtering

Hierarchical GPT with Congruent Transformers for Multi-Sentence Language Models

H3T: Efficient Integration of Memory Optimization and Parallelism for Large-scale Transformer Training.

Alternating Updates for Efficient Transformers

Hierarchical Topic-Aware Contextualized Transformers

Hierarchical Transformers are Efficient Meta-Reinforcement Learners

A Multi-Level Framework for Accelerating Training Transformer Models

Pushdown Layers: Encoding Recursive Structure in Transformer Language Models

H3T: Efficient Integration of Memory Optimization and Parallelism for High-Throughput Transformer Training

Reformer: The Efficient Transformer

Jump to Conclusions: Short-Cutting Transformers With Linear Transformations

Efficiently Scaling Transformer Inference

Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically

Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding

Faster Depth-Adaptive Transformers