Abstract:Click-Through Rate (CTR) prediction is a crucial component in the online advertising industry. In order to produce a personalized CTR prediction, an industry-level CTR prediction model commonly takes a high-dimensional (e.g., 100 or 1000 billions of features) sparse vector (that is encoded from query keywords, user portraits, etc.) as input. As a result, the model requires Terabyte scale parameters to embed the high-dimensional input. Hierarchical distributed GPU parameter server has been proposed to enable GPU with limited memory to train the massive network by leveraging CPU main memory and SSDs as secondary storage. We identify two major challenges in the existing GPU training framework for massive-scale ad models and propose a collection of optimizations to tackle these challenges: (a) the GPU, CPU, SSD rapidly communicate with each other during the training. The connections between GPUs and CPUs are non-uniform due to the hardware topology. The data communication route should be optimized according to the hardware topology; (b) GPUs in different computing nodes frequently communicates to synchronize parameters. We are required to optimize the communications so that the distributed system can become scalable. In this paper, we propose a hardware-aware training workflow that couples the hardware topology into the algorithm design. To reduce the extensive communication between computing nodes, we introduce a $k$-step model merging algorithm for the popular Adam optimizer and provide its convergence rate in non-convex optimization. To the best of our knowledge, this is the first application of $k$-step adaptive optimization method in industrial-level CTR model training. The numerical results on real-world data confirm that the optimized system design considerably reduces the training time of the massive model, with essentially no loss in accuracy.

Failure Tolerant Training with Persistent Memory Disaggregation over CXL

Efficient Fault Tolerance for Recommendation Model Training via Erasure Coding

CXL Memory as Persistent Memory for Disaggregated HPC: A Practical Approach

NDRec: A Near-Data Processing System for Training Large-Scale Recommendation Models

MTrainS: Improving DLRM training efficiency using heterogeneous memories

ECRM: Efficient Fault Tolerance for Recommendation Model Training via Erasure Coding

CXL Shared Memory Programming: Barely Distributed and Almost Persistent

Rcmp: Reconstructing RDMA-Based Memory Disaggregation via CXL

TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models

Mixed-Precision Embedding Using a Cache

Partial Failure Resilient Memory Management System for (Cxl-Based) Distributed Shared Memory

DisaggRec: Architecting Disaggregated Systems for Large-Scale Personalized Recommendation

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

CmnRec: Sequential Recommendations with Chunk-accelerated Memory Network

Reliable and Efficient In-Memory Fault Tolerance of Large Language Model Pretraining

Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices

Kraken: Memory-Efficient Continual Learning for Large-Scale Real-Time Recommendations

Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertising

GCNear: A Hybrid Architecture for Efficient GCN Training with Near-Memory Processing

RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing

Efficient Data-Parallel Continual Learning with Asynchronous Distributed Rehearsal Buffers