Abstract:As Large Language Models (LLMs) achieve remarkable progress in language understanding and generation, their training efficiency has become a critical concern. Traditionally, LLMs are trained to predict the next token in a sequence. Despite the success of token-level training, it suffers from considerable computational costs due to the need to process an extensive number of tokens. To mitigate this issue, this paper introduces patch-level training for LLMs, which reduces the sequence length by compressing multiple tokens into a single patch. During patch-level training, we feed the language model shorter sequences of patches and train it to predict the next patch, thereby processing the majority of the training data at a significantly reduced computational cost. Following this, the model continues token-level training on the remaining training data to align with the inference mode. Experiments on a diverse range of models (370M-2.7B parameters) demonstrate that patch-level training can reduce overall computational costs to 0.5$\times$, without compromising the model performance compared to token-level training. Source code: \url{<a class="link-external link-https" href="https://github.com/shaochenze/PatchTrain" rel="external noopener nofollow">this https URL</a>}.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the high training costs associated with large language models (LLMs). Specifically, as the model capacity increases, the amount of training data required for LLMs also increases, leading to a significant rise in computational costs. This has become a major bottleneck in developing the next generation of LLMs. To solve this problem, the authors propose a new training method called "patch-level training," which significantly reduces training costs by aggregating multiple tokens into a higher information density unit (referred to as a "patch"), without sacrificing model performance. ### Main Contributions 1. **Patch-Level Training Method**: The authors introduce the patch-level training method, where multiple tokens are aggregated into a patch, serving as the basic text unit for training. During patch-level training, the model processes shorter sequences of patches and predicts the next patch, thereby handling most of the training data at a lower cost. 2. **Two-Stage Training Process**: Patch-level training is divided into two stages: first, patch-level training, followed by token-level training. After completing patch-level training, the parameters obtained are used to initialize the token-level model, which continues training on the remaining data to adapt the knowledge gained from patch-level training. 3. **Experimental Validation**: The authors conducted experiments on models of different scales (370M to 2.7B parameters), showing that patch-level training can reduce overall training costs to 0.5 times the original cost without affecting model performance. ### Experimental Results - **Performance Comparison**: Experimental results show that patch-level training not only significantly reduces training costs but also, in some cases, improves model performance. For example, for a 370M parameter Transformer model, patch-level training reduces computational costs by half while achieving better perplexity and zero-shot evaluation accuracy than a model trained from scratch. - **Multiple Rounds of Training**: When training data is limited, patch-level training outperforms traditional token-level training in multiple rounds of training, indicating that this method has advantages in data utilization efficiency. - **Scalability Analysis**: The authors also explore the scalability of patch-level training across different model sizes and data scales. The results show that patch-level training performs better with larger data volumes, but the performance improvement diminishes with larger model sizes. ### Conclusion The patch-level training method provides an effective way to reduce the training costs of large language models, significantly decreasing computational resource consumption while maintaining model performance. This method has important potential for practical applications, especially in resource-constrained scenarios.

Patch-Level Training for Large Language Models

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention

Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale

Aggressive Post-Training Compression on Extremely Large Language Models

Training-Free Long-Context Scaling of Large Language Models

LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models

PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patches

Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum

Super Tiny Language Models

On the Effectiveness of Incremental Training of Large Language Models

Simple and Scalable Strategies to Continually Pre-train Large Language Models

A Learning Rate Path Switching Training Paradigm for Version Updates of Large Language Models

Sparsity-Accelerated Training for Large Language Models

Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models

Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model

Learn To be Efficient: Build Structured Sparsity in Large Language Models

Distributed Training of Large Language Models

A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs

Scaling Law for Language Models Training Considering Batch Size

Optimizing Low-Resource Language Model Training: Comprehensive Analysis of Multi-Epoch, Multi-Lingual, and Two-Stage Approaches