Abstract:Large language models (LLMs) have demonstrated remarkable success as foundational models, benefiting various downstream applications through fine-tuning. Recent studies on loss scaling have demonstrated the superior performance of larger LLMs compared to their smaller counterparts. Nevertheless, training LLMs with billions of parameters poses significant challenges and requires considerable computational resources. For example, training a one trillion parameter GPT-style model on 20 trillion tokens requires a staggering 120 million exaflops of computation. This research explores efficient distributed training strategies to extract this computation from Frontier, the world's first exascale supercomputer dedicated to open science. We enable and investigate various model and data parallel training techniques, such as tensor parallelism, pipeline parallelism, and sharded data parallelism, to facilitate training a trillion-parameter model on Frontier. We empirically assess these techniques and their associated parameters to determine their impact on memory footprint, communication latency, and GPU's computational efficiency. We analyze the complex interplay among these techniques and find a strategy to combine them to achieve high throughput through hyperparameter tuning. We have identified efficient strategies for training large LLMs of varying sizes through empirical analysis and hyperparameter tuning. For 22 Billion, 175 Billion, and 1 Trillion parameters, we achieved GPU throughputs of $38.38\%$, $36.14\%$, and $31.96\%$, respectively. For the training of the 175 Billion parameter model and the 1 Trillion parameter model, we achieved $100\%$ weak scaling efficiency on 1024 and 3072 MI250X GPUs, respectively. We also achieved strong scaling efficiencies of $89\%$ and $87\%$ for these two models.

Comparative Study of Large Language Model Architectures on Frontier

Optimizing Distributed Training on Frontier for Large Language Models

Evaluation of pre-training large language models on leadership-class supercomputers

Exploring the Potential of Large Language Models in Molecular Tasks: An Insightful Evaluation with GPT‐4

Large Language Models as Master Key: Unlocking the Secrets of Materials Science with GPT

The Future of Molecular Studies Through the Lens of Large Language Models.

Unleashing the Potential of LLMs for Quantum Computing: A Study in Quantum Architecture Design

Mining experimental data from Materials Science literature with Large Language Models: an evaluation study

OMPGPT: A Generative Pre-trained Transformer Model for OpenMP

Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models

Large Language Models on Small Resource-Constrained Systems: Performance Characterization, Analysis and Trade-offs

A Comprehensive Performance Study of Large Language Models on Novel AI Accelerators

Large Language Models (LLMs): Deployment, Tokenomics and Sustainability

HPC-GPT: Integrating Large Language Model for High-Performance Computing

The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4

Large Language Models as Data Preprocessors

Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysis

Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models

A Comparative Analysis of Distributed Training Strategies for GPT-2

Can Large Language Models Learn the Physics of Metamaterials? An Empirical Study with ChatGPT