Abstract:State-of-the-art results in large language models (LLMs) often rely on scale, which becomes computationally expensive. This has sparked a research agenda to reduce these models' parameter count and computational costs without significantly impacting their performance. Our study focuses on transformer-based LLMs, specifically targeting the computationally intensive feedforward networks (FFN), which are less studied than attention blocks. We consider three candidate linear layer approximations in the FFN by combining efficient low-rank and block-diagonal matrices. In contrast to many previous works that examined these approximations, our study i) explores these structures from the training-from-scratch perspective, ii) scales up to 1.3B parameters, and iii) is conducted within recent Transformer-based LLMs rather than convolutional architectures. We first demonstrate they can lead to actual computational gains in various scenarios, including online decoding when using a pre-merge technique. Additionally, we propose a novel training regime, called \textit{self-guided training}, aimed at improving the poor training dynamics that these approximations exhibit when used from initialization. Experiments on the large RefinedWeb dataset show that our methods are both efficient and effective for training and inference. Interestingly, these structured FFNs exhibit steeper scaling curves than the original models. Further applying self-guided training to the structured matrices with 32\% FFN parameters and 2.5$\times$ speed-up enables only a 0.4 perplexity increase under the same training FLOPs. Finally, we develop the wide and structured networks surpassing the current medium-sized and large-sized Transformer in perplexity and throughput performance. Our code is available at \url{<a class="link-external link-https" href="https://github.com/CLAIRE-Labo/StructuredFFN/tree/main" rel="external noopener nofollow">this https URL</a>}.

Leveraging Neighbor Attention Initialization (NAI) for Efficient Training of Pretrained LLMs

GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost

bert2BERT: Towards Reusable Pretrained Language Models

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention

Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies

Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models

LERT: A Linguistically-motivated Pre-trained Language Model

Deep Fusion: Efficient Network Training via Pre-trained Initializations

Reusing Pretrained Models by Multi-linear Operators for Efficient Training

A Compact Pretraining Approach for Neural Language Models

Large Language Model Compression with Neural Architecture Search

Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale

NarrowBERT: Accelerating Masked Language Model Pretraining and Inference

A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs

CPM-2: Large-scale Cost-effective Pre-trained Language Models

Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers

Preparing Lessons for Progressive Training on Language Models

Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models

NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework

APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference

Sparsity-Accelerated Training for Large Language Models