Abstract:State-of-the-art results in large language models (LLMs) often rely on scale, which becomes computationally expensive. This has sparked a research agenda to reduce these models' parameter count and computational costs without significantly impacting their performance. Our study focuses on transformer-based LLMs, specifically targeting the computationally intensive feedforward networks (FFN), which are less studied than attention blocks. We consider three candidate linear layer approximations in the FFN by combining efficient low-rank and block-diagonal matrices. In contrast to many previous works that examined these approximations, our study i) explores these structures from the training-from-scratch perspective, ii) scales up to 1.3B parameters, and iii) is conducted within recent Transformer-based LLMs rather than convolutional architectures. We first demonstrate they can lead to actual computational gains in various scenarios, including online decoding when using a pre-merge technique. Additionally, we propose a novel training regime, called \textit{self-guided training}, aimed at improving the poor training dynamics that these approximations exhibit when used from initialization. Experiments on the large RefinedWeb dataset show that our methods are both efficient and effective for training and inference. Interestingly, these structured FFNs exhibit steeper scaling curves than the original models. Further applying self-guided training to the structured matrices with 32\% FFN parameters and 2.5$\times$ speed-up enables only a 0.4 perplexity increase under the same training FLOPs. Finally, we develop the wide and structured networks surpassing the current medium-sized and large-sized Transformer in perplexity and throughput performance. Our code is available at \url{<a class="link-external link-https" href="https://github.com/CLAIRE-Labo/StructuredFFN/tree/main" rel="external noopener nofollow">this https URL</a>}.

Freely Long-Thinking Transformer (FraiLT)

FIT: Far-reaching Interleaved Transformers

Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator

Reformer: The Efficient Transformer

Longformer: The Long-Document Transformer

Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers

Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices

Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers

Trainable Transformer in Transformer

BEExformer: A Fast Inferencing Transformer Architecture via Binarization with Multiple Early Exits

Focused Transformer: Contrastive Training for Context Scaling

Fast-and-Frugal Text-Graph Transformers are Effective Link Predictors

Unlimiformer: Long-Range Transformers with Unlimited Length Input

Resformer: Combine quadratic linear transformation with efficient sparse Transformer for long-term series forecasting

Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting

Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis

FTRANS: Energy-Efficient Acceleration of Transformers using FPGA

Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA

Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Traces

Lite Transformer with Long-Short Range Attention

Do Efficient Transformers Really Save Computation?