Abstract:The development of large language models (LLMs) has been instrumental in advancing state-of-the-art natural language processing applications. Training LLMs with billions of parameters and trillions of tokens require sophisticated distributed systems that enable composing and comparing several state-of-the-art techniques in order to efficiently scale across thousands of accelerators. However, existing solutions are complex, scattered across multiple libraries/repositories, lack interoperability, and are cumbersome to maintain. Thus, curating and empirically comparing training recipes require non-trivial engineering effort. This paper introduces TorchTitan, an open-source, PyTorch-native distributed training system that unifies state-of-the-art techniques, streamlining integration and reducing overhead. TorchTitan enables 3D parallelism in a modular manner with elastic scaling, providing comprehensive logging, checkpointing, and debugging tools for production-ready training. It also incorporates hardware-software co-designed solutions, leveraging features like Float8 training and SymmetricMemory. As a flexible test bed, TorchTitan facilitates custom recipe curation and comparison, allowing us to develop optimized training recipes for Llama 3.1 and provide guidance on selecting techniques for maximum efficiency based on our experiences. We thoroughly assess TorchTitan on the Llama 3.1 family of LLMs, spanning 8 billion to 405 billion parameters, and showcase its exceptional performance, modular composability, and elastic scalability. By stacking training optimizations, we demonstrate accelerations of 65.08% with 1D parallelism at the 128-GPU scale (Llama 3.1 8B), an additional 12.59% with 2D parallelism at the 256-GPU scale (Llama 3.1 70B), and an additional 30% with 3D parallelism at the 512-GPU scale (Llama 3.1 405B) on NVIDIA H100 GPUs over optimized baselines.

TorchCP: A Python Library for Conformal Prediction

Conformal Prediction: A Data Perspective

A novel Deep Learning approach for one-step Conformal Prediction approximation

Predictive Inference With Fast Feature Conformal Prediction

Provably Robust Conformal Prediction with Improved Efficiency

Efficient Conformal Prediction via Cascaded Inference with Expanded Admission

Robust Yet Efficient Conformal Prediction Sets

Verifiably Robust Conformal Prediction

Improving Uncertainty Quantification of Deep Classifiers via Neighborhood Conformal Prediction: Novel Algorithm and Theoretical Analysis

Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python

Conformal Prediction Regions for Time Series using Linear Complementarity Programming

OpTorch: Optimized deep learning architectures for resource limited environments

Conformal Loss-Controlling Prediction

On Temperature Scaling and Conformal Prediction of Deep Classifiers

An Information Theoretic Perspective on Conformal Prediction

Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration

Conformal Prediction: A Gentle Introduction

TorchOpt: an Efficient Library for Differentiable Optimization.

TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training

Efficient Conformal Prediction under Data Heterogeneity

Predictive Inference with Feature Conformal Prediction