Abstract:Deep Learning is a very powerful machine learning model. Deep Learning trains a large number of parameters for multiple layers and is very slow when data is in large scale and the architecture size is large. Inspired from the shrinking technique used in accelerating computation of Support Vector Machines (SVM) algorithm and screening technique used in LASSO, we propose a shrinking Deep Learning with recall (sDLr) approach to speed up deep learning computation. We experiment shrinking Deep Learning with recall (sDLr) using Deep Neural Network (DNN), Deep Belief Network (DBN) and Convolution Neural Network (CNN) on 4 data sets. Results show that the speedup using shrinking Deep Learning with recall (sDLr) can reach more than 2.0 while still giving competitive classification performance.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that when deep - learning algorithms handle large - scale data and large - network architectures, the training process has high computational costs and takes too long. Specifically: 1. **Computational complexity of deep learning**: Deep - learning models need to train a large number of parameters. Especially in the case of large - scale data and complex network architectures, the training process is very slow. 2. **Need for accelerated computing**: Inspired by the shrinking technique in support vector machines (SVM) and the screening technique in LASSO, the author proposes a new method - shrinking Deep Learning with Recall (sDLr) - to accelerate the computation of deep learning. 3. **Accelerating training while maintaining classification performance**: The goal of the paper is to significantly reduce the running time while still maintaining competitive classification performance. Verified by experiments, the sDLr method can achieve more than a two - fold acceleration on multiple deep - learning models (such as DNN, DBN, and CNN), and can also improve classification accuracy on some datasets. ### Specific problem description - **Low computational efficiency**: Due to its multi - layer non - linear structure and a large number of parameters, the deep - learning model is very time - consuming in the training process when handling large - scale data. - **High resource consumption**: The training of deep - learning models not only requires a large amount of computational resources but also takes a long time, which poses a challenge to practical applications. ### Solution - **Shrinking technique**: By removing samples that have less influence on the gradient, the amount of data participating in training in each iteration is reduced, thereby accelerating the training process. - **Recall mechanism**: When the number of remaining training samples is reduced to a certain extent, all training samples are re - introduced to ensure that the generalization ability of the model is not affected. Through these methods, the paper proposes a new method that can accelerate training without significantly degrading the model performance.

Accelerating Deep Learning with Shrinkage and Recall

Deep Neural Network Acceleration with Sparse Prediction Layers

Learning Slimming SSD Through Pruning and Knowledge Distillation

Inshrinkerator: Compressing Deep Learning Training Checkpoints via Dynamic Quantization

Cloning, expression and biochemical characterisation of a unique thermostable pullulan-hydrolysing enzyme from the hyperthermophilic archaeon Thermococcus aggregans.

Improving deep neural networks for LVCSR using dropout and shrinking structure

Learning Instance-wise Sparsity for Accelerating Deep Models

DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

A MIC-based acceleration model of Deep Learning

Large-Scale Deep Learning Optimizations: A Comprehensive Survey

Data Shunt: Collaboration of Small and Large Models for Lower Costs and Better Performance

DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices

Learning Efficient Convolutional Networks Through Network Slimming.

Channel Pruning for Accelerating Very Deep Neural Networks

Accelerating Very Deep Convolutional Networks for Classification and Detection

S2 Reducer: High-Performance Sparse Communication to Accelerate Distributed Deep Learning

Log-sum enhanced sparse deep neural network

Enabling Flexible Resource Allocation in Mobile Deep Learning Systems

A novel genetic algorithm-based approach for compression and acceleration of deep learning convolution neural network: an application in computer tomography lung cancer data

Minimizing Area and Energy of Deep Learning Hardware Design Using Collective Low Precision and Structured Compression