Abstract:Training modern LLMs is extremely resource intensive, and customizing them for various deployment scenarios characterized by limited compute and memory resources through repeated training is impractical. In this paper, we introduce Flextron, a network architecture and post-training model optimization framework supporting flexible model deployment. The Flextron architecture utilizes a nested elastic structure to rapidly adapt to specific user-defined latency and accuracy targets during inference with no additional fine-tuning required. It is also input-adaptive, and can automatically route tokens through its sub-networks for improved performance and efficiency. We present a sample-efficient training method and associated routing algorithms for systematically transforming an existing trained LLM into a Flextron model. We evaluate Flextron on the GPT-3 and LLama-2 family of LLMs, and demonstrate superior performance over multiple end-to-end trained variants and other state-of-the-art elastic networks, all with a single pretraining run that consumes a mere 7.63% tokens compared to original pretraining.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: the efficiency and performance issues faced by modern large - scale language models (LLMs) when deployed in resource - constrained environments. Specifically, since these models usually contain billions of parameters, it is difficult for them to operate efficiently in scenarios with limited computing resources and memory. To solve this problem, the author proposes a new architecture and optimization framework - FLEXTRON. ### Goals of FLEXTRON 1. **Flexibly adapt to different latency and accuracy requirements**: - The FLEXTRON architecture utilizes a nested elastic structure to quickly adapt to specific user - defined latency and accuracy goals during the inference process without additional fine - tuning. 2. **Input adaptation**: - FLEXTRON can automatically select the optimal sub - network according to the input data, thereby improving performance and efficiency. 3. **Efficient training method**: - A sample - efficient training method and a routing algorithm are proposed, which can convert the existing pre - trained LLM into a FLEXTRON model with only one pre - training, and the amount of training data consumed is only 7.63% of the original pre - training. ### Main contributions 1. **Novel architecture**: - FLEXTRON can flexibly adapt to different latency and accuracy goals during the inference process without additional fine - tuning. 2. **Post - training optimization framework**: - Systematically convert the existing pre - trained LLM into a dynamic (input - adaptive) elastic network. 3. **New routing algorithm**: - Static and dynamic routing algorithms are introduced, which can automatically select the optimal sub - network according to the latency goal and/or input tokens, and a new surrogate model is proposed for effectively training the router. 4. **Efficient sampling training method**: - This method significantly reduces the training computation of the elastic network. Through these improvements, FLEXTRON not only improves the deployment efficiency of large - scale language models in resource - constrained environments, but also maintains high performance and accuracy.

Flextron: Many-in-One Flexible Large Language Model

FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning

FlexModel: A Framework for Interpretability of Distributed Large Language Models

FlexTrain: A Dynamic Training Framework for Heterogeneous Devices Environments

FlashFlex: Accommodating Large Language Model Training over Heterogeneous Environment

Flexora: Flexible Low Rank Adaptation for Large Language Models

Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices

Tutel: Adaptive Mixture-of-Experts at Scale

FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU

FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement

High-throughput Generative Inference of Large Language Models with a Single GPU

FinGPT-HPC: Efficient Pretraining and Finetuning Large Language Models for Financial Applications with High-Performance Computing

Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources

FoldGPT: Simple and Effective Large Language Model Compression Scheme

Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation

FlexFL: Heterogeneous Federated Learning via APoZ-Guided Flexible Pruning in Uncertain Scenarios

Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization

One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments

Efficient and Economic Large Language Model Inference with Attention Offloading

AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment