Abstract:The power of large language models (LLMs) has been demonstrated through numerous data and computing resources. However, the application of language models on mobile devices is facing huge challenge on the computation and memory costs, that is, tiny language models with high performance are urgently required. Limited by the highly complex training process, there are many details for optimizing language models that are seldom studied carefully. In this study, based on a tiny language model with 1B parameters, we carefully design a series of empirical study to analyze the effect of each component. Three perspectives are mainly discussed, \ie, neural architecture, parameter initialization, and optimization strategy. Several design formulas are empirically proved especially effective for tiny language models, including tokenizer compression, architecture tweaking, parameter inheritance and multiple-round training. Then we train PanGu-$\pi$-1B Pro and PanGu-$\pi$-1.5B Pro on 1.6T multilingual corpora, following the established formulas. Experimental results demonstrate the improved optimization and architecture yield a notable average improvement of 8.87 on benchmark evaluation sets for PanGu-$\pi$-1B Pro. Besides, PanGu-$\pi$-1.5B Pro surpasses a range of SOTA models with larger model sizes, validating its superior performance. The code is available at

What problem does this paper attempt to address?

This paper mainly discusses how to optimize and design small-scale language models to address the high computational and memory costs, as well as the high performance requirements, when applied on mobile devices. The research is based on a small-scale language model with 1 billion parameters, and analyzes the effects of various components through a series of empirical studies, focusing on three aspects: neural architecture, parameter initialization, and model optimization. 1. **Neural Architecture**: The paper points out that inheriting the tokenizer directly from large-scale models will lead to redundant parameters and increase computational overhead. Compressing the tokenizer by removing low-frequency vocabulary can improve the representation efficiency of the model. The depth, width, and expansion rate of the feed-forward network (FFN) have a significant impact on performance, with depth being particularly important for small-scale models but sacrificing inference speed. 2. **Parameter Initialization**: Inheriting parameters from large-scale models can improve performance and accelerate convergence. Identifying key parameters is important, especially the layers near the beginning and end of the model are more important than intermediate layers. Within each layer, data-driven learnable norms are more effective than heuristic methods. 3. **Model Optimization**: Small-scale models are more prone to forgetting data compared to large-scale models, and multi-round training helps enhance memory. A simple sample selection strategy is proposed to alleviate the cost of multi-round training, and the relationship between batch size and learning rate for small-scale models is studied. The paper develops the PanGu-π-1B Pro and PanGu-π-1.5B Pro models, which achieve average performance improvements of 8.87 and surpass the SOTA model Qwen-1.8B with more parameters, respectively, through improved architecture and optimization methods. These findings emphasize the importance of customized optimization for small-scale language models to achieve efficient and high-performance models.

Rethinking Optimization and Architecture for Tiny Language Models

MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensation

YuLan-Mini: An Open Data-efficient Language Model

Super Tiny Language Models

BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices

Large Language Models on Mobile Devices: Measurements, Analysis, and Insights

PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing

TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones

Small Language Models: Survey, Measurements, and Insights

Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models

PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training

TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers

Towards Optimizing with Large Language Models

Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations

TinyLlama: An Open-Source Small Language Model

A Method to Build a Super Small but Practically Accurate Language Model for Handheld Devices

Search for Efficient Large Language Models

Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks

A Study of Optimizations for Fine-tuning Large Language Models

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation