Abstract:Click-Through Rate (CTR) prediction is a crucial component in the online advertising industry. In order to produce a personalized CTR prediction, an industry-level CTR prediction model commonly takes a high-dimensional (e.g., 100 or 1000 billions of features) sparse vector (that is encoded from query keywords, user portraits, etc.) as input. As a result, the model requires Terabyte scale parameters to embed the high-dimensional input. Hierarchical distributed GPU parameter server has been proposed to enable GPU with limited memory to train the massive network by leveraging CPU main memory and SSDs as secondary storage. We identify two major challenges in the existing GPU training framework for massive-scale ad models and propose a collection of optimizations to tackle these challenges: (a) the GPU, CPU, SSD rapidly communicate with each other during the training. The connections between GPUs and CPUs are non-uniform due to the hardware topology. The data communication route should be optimized according to the hardware topology; (b) GPUs in different computing nodes frequently communicates to synchronize parameters. We are required to optimize the communications so that the distributed system can become scalable. In this paper, we propose a hardware-aware training workflow that couples the hardware topology into the algorithm design. To reduce the extensive communication between computing nodes, we introduce a $k$-step model merging algorithm for the popular Adam optimizer and provide its convergence rate in non-convex optimization. To the best of our knowledge, this is the first application of $k$-step adaptive optimization method in industrial-level CTR model training. The numerical results on real-world data confirm that the optimized system design considerably reduces the training time of the massive model, with essentially no loss in accuracy.

Mobius: Fine Tuning Large-Scale Models on Commodity GPU Servers

MOC: Multi-Objective Mobile CPU-GPU Co-Optimization for Power-Efficient DNN Inference

Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression

On Optimizing the Communication of Model Parallelism

An Optimization Toolchain Design Of Deep Learning Deployment Based On Heterogeneous Computing Platform

Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers

An Efficient 2D Method for Training Super-Large Deep Learning Models

Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertising

SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Offline and Online Parallelization.

Optimizing execution for pipelined‐based distributed deep learning in a heterogeneously networked GPU cluster

MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud

HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading

MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs

Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms

Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling

Efficient Large-Scale Language Model Training on GPU Clusters

An Analysis of Collocation on GPUs for Deep Learning Training

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models

Inference Performance Optimization for Large Language Models on CPUs