Abstract:State-of-the-art results in large language models (LLMs) often rely on scale, which becomes computationally expensive. This has sparked a research agenda to reduce these models' parameter count and computational costs without significantly impacting their performance. Our study focuses on transformer-based LLMs, specifically targeting the computationally intensive feedforward networks (FFN), which are less studied than attention blocks. We consider three candidate linear layer approximations in the FFN by combining efficient low-rank and block-diagonal matrices. In contrast to many previous works that examined these approximations, our study i) explores these structures from the training-from-scratch perspective, ii) scales up to 1.3B parameters, and iii) is conducted within recent Transformer-based LLMs rather than convolutional architectures. We first demonstrate they can lead to actual computational gains in various scenarios, including online decoding when using a pre-merge technique. Additionally, we propose a novel training regime, called \textit{self-guided training}, aimed at improving the poor training dynamics that these approximations exhibit when used from initialization. Experiments on the large RefinedWeb dataset show that our methods are both efficient and effective for training and inference. Interestingly, these structured FFNs exhibit steeper scaling curves than the original models. Further applying self-guided training to the structured matrices with 32\% FFN parameters and 2.5$\times$ speed-up enables only a 0.4 perplexity increase under the same training FLOPs. Finally, we develop the wide and structured networks surpassing the current medium-sized and large-sized Transformer in perplexity and throughput performance. Our code is available at \url{<a class="link-external link-https" href="https://github.com/CLAIRE-Labo/StructuredFFN/tree/main" rel="external noopener nofollow">this https URL</a>}.

Inheritune: Training Smaller Yet More Attentive Language Models

Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models

Towards smaller, faster decoder-only transformers: Architectural variants and their implications

Adaptive Large Language Models By Layerwise Attention Shortcuts

Why Larger Language Models Do In-context Learning Differently?

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

Evolving Subnetwork Training for Large Language Models

Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA

TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones

Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations

From pre-training to fine-tuning: An in-depth analysis of Large Language Models in the biomedical domain

LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Models

Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers

TAIA: Large Language Models are Out-of-Distribution Data Learners

MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models

Tuning Language Models by Mixture-of-Depths Ensemble

MindLLM: Lightweight Large Language Model Pre-Training, Evaluation and Domain Application

Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE

Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models