Abstract:Large pre-trained neural language models have brought immense progress to both NLP and software engineering. Models in OpenAI's GPT series now dwarf Google's BERT and Meta's RoBERTa, which previously set new benchmarks on a wide range of NLP applications. These models are trained on massive corpora of heterogeneous data from web crawls, which enables them to learn general language patterns and semantic relationships. However, the largest models are both expensive to train and deploy and are often closed-source, so we lack access to their data and design decisions. We argue that this trend towards large, general-purpose models should be complemented with single-purpose, more modestly sized pre-trained models. In this work, we take StackOverflow (SO) as a domain example in which large volumes of rich aligned code and text data is available. We adopt standard practices for pre-training large language models, including using a very large context size (2,048 tokens), batch size (0.5M tokens) and training set (27B tokens), coupled with a powerful toolkit (Megatron-LM), to train two models: SOBertBase, with 109M parameters, and SOBertLarge with 762M parameters, at a budget of just $\$187$ and $\$800$ each. We compare the performance of our models with both the previous SOTA model trained on SO data exclusively as well general-purpose BERT models and OpenAI's ChatGPT on four SO-specific downstream tasks - question quality prediction, closed question prediction, named entity recognition and obsoletion prediction (a new task we introduce). Not only do our models consistently outperform all baselines, the smaller model is often sufficient for strong results. Both models are released to the public. These results demonstrate that pre-training both extensively and properly on in-domain data can yield a powerful and affordable alternative to leveraging closed-source general-purpose models.

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Efficient Large-Scale Language Model Training on GPU Clusters

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models

Optimizing Distributed Training on Frontier for Large Language Models

A Multi-Level Framework for Accelerating Training Transformer Models

PaLM: Scaling Language Modeling with Pathways

Training Compute-Optimal Large Language Models

OMPGPT: A Generative Pre-trained Transformer Model for OpenMP

TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models

Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models

A Comparative Analysis of Distributed Training Strategies for GPT-2

An Efficient 2D Method for Training Super-Large Deep Learning Models

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

FPM: A Collection of Large-scale Foundation Pre-trained Language Models

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

"Medium" LMs of Code in the Era of LLMs: Lessons From StackOverflow

Accelerating the Training of Large Language Models Using Efficient Activation Rematerialization and Optimal Hybrid Parallelism.

Blockwise Parallel Transformer for Large Context Models

Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training

bert2BERT: Towards Reusable Pretrained Language Models

Optimizing Multi-Task Learning for Enhanced Performance in Large Language Models