Abstract:The field of deep learning has witnessed significant progress, particularly in computer vision (CV), natural language processing (NLP), and speech. The use of large-scale models trained on vast amounts of data holds immense promise for practical applications, enhancing industrial productivity and facilitating social development. With the increasing demands on computational capacity, though numerous studies have explored the efficient training, a comprehensive summarization on acceleration techniques of training deep learning models is still much anticipated. In this survey, we present a detailed review for training acceleration. We consider the fundamental update formulation and split its basic components into five main perspectives: (1) data-centric: including dataset regularization, data sampling, and data-centric curriculum learning techniques, which can significantly reduce the computational complexity of the data samples; (2) model-centric, including acceleration of basic modules, compression training, model initialization and model-centric curriculum learning techniques, which focus on accelerating the training via reducing the calculations on parameters; (3) optimization-centric, including the selection of learning rate, the employment of large batchsize, the designs of efficient objectives, and model average techniques, which pay attention to the training policy and improving the generality for the large-scale models; (4) budgeted training, including some distinctive acceleration methods on source-constrained situations; (5) system-centric, including some efficient open-source distributed libraries/systems which provide adequate hardware support for the implementation of acceleration algorithms. By presenting this comprehensive taxonomy, our survey presents a comprehensive review to understand the general mechanisms within each component and their joint interaction.

A MIC-based acceleration model of Deep Learning

MLCNN: Cross-Layer Cooperative Optimization and Accelerator Architecture for Speeding Up Deep Learning Applications

Close the Gap Between Deep Learning and Mobile Intelligence by Incorporating Training in the Loop

Explore Training of Deep Convolutional Neural Networks on Battery-powered Mobile Devices: Design and Application

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

Accelerating Deep Learning Systems Via Critical Set Identification and Model Compression.

Research on Deep Learning Acceleration Technique

Hardware Accelerated Optimization of Deep Learning Model on Artificial Intelligence Chip

Parallel Computing In Dnns Using Cpu And Mic

On Efficient Training of Large-Scale Deep Learning Models: A Literature Review

An Acceleration Method Based on Deep Learning and Multilinear Feature Space

Advanced Deep Neural Network Technique for Microwave Parametric Modeling

Training One DeePMD Model in Minutes: a Step Towards Online Learning.

Accelerating Neural Network Training: A Brief Review

Training Acceleration for Deep Neural Networks: A Hybrid Parallelization Strategy

Optimized Monocular Depth Estimation with Reparameterization on Embedded Devices.

EncodingNet: A Novel Encoding-based MAC Design for Efficient Neural Network Acceleration

Large Language Model Inference Acceleration Based on Hybrid Model Branch Prediction

Efficient Deep Learning Inference Based on Model Compression.

RRAM based learning acceleration.