Abstract:The field of deep learning has witnessed significant progress, particularly in computer vision (CV), natural language processing (NLP), and speech. The use of large-scale models trained on vast amounts of data holds immense promise for practical applications, enhancing industrial productivity and facilitating social development. With the increasing demands on computational capacity, though numerous studies have explored the efficient training, a comprehensive summarization on acceleration techniques of training deep learning models is still much anticipated. In this survey, we present a detailed review for training acceleration. We consider the fundamental update formulation and split its basic components into five main perspectives: (1) data-centric: including dataset regularization, data sampling, and data-centric curriculum learning techniques, which can significantly reduce the computational complexity of the data samples; (2) model-centric, including acceleration of basic modules, compression training, model initialization and model-centric curriculum learning techniques, which focus on accelerating the training via reducing the calculations on parameters; (3) optimization-centric, including the selection of learning rate, the employment of large batchsize, the designs of efficient objectives, and model average techniques, which pay attention to the training policy and improving the generality for the large-scale models; (4) budgeted training, including some distinctive acceleration methods on source-constrained situations; (5) system-centric, including some efficient open-source distributed libraries/systems which provide adequate hardware support for the implementation of acceleration algorithms. By presenting this comprehensive taxonomy, our survey presents a comprehensive review to understand the general mechanisms within each component and their joint interaction.

Research on Deep Learning Acceleration Technique

Deep Learning on Mobile and Embedded Devices: State-of-the-art, Challenges, and Future Directions

Application of Deep Learning in Back-End Simulation: Challenges and Opportunities

Hardware Accelerated Optimization of Deep Learning Model on Artificial Intelligence Chip

On Efficient Training of Large-Scale Deep Learning Models: A Literature Review

Performance and Power Evaluation of AI Accelerators for Training Deep Learning Models

DISTRIBUTED HIGH-PERFORMANCE COMPUTING METHODS FOR ACCELERATING DEEP LEARNING TRAINING

Deep Learning Hardware Acceleration Based on General Vector DSP

Research on Parallel Acceleration for Deep Learning Inference Based on Many-Core ARM Platform.

A Quick Survey on Large Scale Distributed Deep Learning Systems.

A Survey of Accelerator Architectures for Deep Neural Networks

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

Efficient Hardware Optimization Strategies For Deep Neural Networks Acceleration Chip

Model Compression and Acceleration for Deep Neural Networks: the Principles, Progress, and Challenges

A Survey on Acceleration of Deep Convolutional Neural Networks

Accelerating Deep Neural Networks implementation: A survey

Towards Ultra-High Performance and Energy Efficiency of Deep Learning Systems: An Algorithm-Hardware Co-Optimization Framework

An Overview of FPGA Based Deep Learning Accelerators: Challenges and Opportunities.

A Survey of FPGA Based Deep Learning Accelerators: Challenges and Opportunities

Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey

A MIC-based acceleration model of Deep Learning