Abstract:Click-Through Rate (CTR) prediction is a crucial component in the online advertising industry. In order to produce a personalized CTR prediction, an industry-level CTR prediction model commonly takes a high-dimensional (e.g., 100 or 1000 billions of features) sparse vector (that is encoded from query keywords, user portraits, etc.) as input. As a result, the model requires Terabyte scale parameters to embed the high-dimensional input. Hierarchical distributed GPU parameter server has been proposed to enable GPU with limited memory to train the massive network by leveraging CPU main memory and SSDs as secondary storage. We identify two major challenges in the existing GPU training framework for massive-scale ad models and propose a collection of optimizations to tackle these challenges: (a) the GPU, CPU, SSD rapidly communicate with each other during the training. The connections between GPUs and CPUs are non-uniform due to the hardware topology. The data communication route should be optimized according to the hardware topology; (b) GPUs in different computing nodes frequently communicates to synchronize parameters. We are required to optimize the communications so that the distributed system can become scalable. In this paper, we propose a hardware-aware training workflow that couples the hardware topology into the algorithm design. To reduce the extensive communication between computing nodes, we introduce a $k$-step model merging algorithm for the popular Adam optimizer and provide its convergence rate in non-convex optimization. To the best of our knowledge, this is the first application of $k$-step adaptive optimization method in industrial-level CTR model training. The numerical results on real-world data confirm that the optimized system design considerably reduces the training time of the massive model, with essentially no loss in accuracy.

Large-Scale Training System for 100-Million Classification at Alibaba

KunPeng: Parameter Server Based Distributed Learning Systems and Its Applications in Alibaba and Ant Financial

Large Scale Long-tailed Product Recognition System at Alibaba

Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent

Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

On Distributed Deep Network for Processing Large-Scale Sets of Complex Data

A Parallel Adaboost-Backpropagation Neural Network for Massive Image Dataset Classification

Multi-task Cascade Deep Convolutional Neural Networks for Large-Scale Commodity Recognition

An Efficient 2D Method for Training Super-Large Deep Learning Models

Deep Multi-task Learning for Large-Scale Image Classification

Training Deep Neural Networks with Large-scale Datasets on Sunway High Performance Computer

An Efficient Training Approach for Very Large Scale Face Recognition

Massively scalable prototype learning for heterogeneous parallel computing architecture

Large Scale Language Modeling: Converging on 40GB of Text in Four Hours

Improved classification approach for use with large-scale scene images in the Hadoop cluster environment.

High-Performance Machine Learning for Large-Scale Data Classification Considering Class Imbalance

CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPU

Distributed Learning for Large-Scale Models at Edge With Privacy Protection

Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertising