What problem does this paper attempt to address?

The main problem that this paper attempts to solve is: how to optimize the training methods of neural networks for approximate hardware. Specifically, the authors point out that although approximate computing methods show great potential in deep learning, especially in reducing cost and power consumption when performing inference tasks on battery - powered devices, these methods have not yet fully realized their potential due to the lack of specialized training methods. ### Background and Problem Description of the Paper 1. **Potential of Approximate Computing Methods** - Approximate computing methods (such as stochastic computing, approximate arithmetic, and analog computing) improve performance by sacrificing some computational precision. - These methods are particularly suitable for inference tasks on battery - powered devices with power budget limitations. 2. **Existing Challenges** - At present, most research focuses on approximate computing in the inference stage and ignores the training stage. - Since the errors introduced by approximate computing will affect the accuracy of the model, it is difficult to directly use the models trained for floating - point or fixed - point computing. - The lack of training methods specifically for approximate hardware leads to a limited application range of approximate computing, which is usually only applicable to simple models and datasets, or requires sacrificing performance to maintain accuracy. ### Objectives of the Paper To overcome the above challenges, this paper proposes the following objectives: - **Develop Specialized Training Methods**: Enable neural networks to be efficiently trained on approximate hardware while maintaining high accuracy. - **Accelerate the Training Process**: Propose new techniques to significantly reduce training time, such as through activation function approximation, error injection, and gradient checkpointing methods. ### Main Contributions The main contributions of the paper include: 1. **Activation Function Approximation**: Use activation functions to approximate computational errors during the back - propagation process. 2. **Error Injection**: Reduce the time of each training iteration by introducing errors and combine accurate modeling to maintain model accuracy. 3. **Gradient Checkpointing**: Reduce memory consumption through the gradient checkpointing technique, thereby supporting larger - batch training and improving GPU utilization. 4. **Significantly Accelerate Training**: Experiments show that the proposed training method can shorten the end - to - end training time by up to 18 times, making it possible to train complex models on a single consumer - level GPU. ### Conclusion Through these improvements, the paper demonstrates how to effectively optimize the training of neural networks for approximate hardware, thereby promoting the further development of approximate computing in practical applications.

Training Neural Networks for Execution on Approximate Hardware

INA: Incremental Network Approximation Algorithm for Limited Precision Deep Neural Networks

Deep Neural Network Approximation for Custom Hardware

AxTrain

ApproxTrain: Fast Simulation of Approximate Multipliers for DNN Training and Inference

Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey

A Hardware/Software Co-Design Methodology for Adaptive Approximate Computing in Clustering and ANN Learning

Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators

ALWANN: Automatic Layer-Wise Approximation of Deep Neural Network Accelerators without Retraining

Concrete: A Per-layer Configurable Framework for Evaluating DNN with Approximate Operators

Efficient Approximate Floating-Point Multiplier With Runtime Reconfigurable Frequency and Precision

Hardware-Aware Softmax Approximation for Deep Neural Networks

Neural Approximating Architecture Targeting Multiple Application Domains

Ax-BxP: Approximate Blocked Computation for Precision-Reconfigurable Deep Neural Network Acceleration

Energy-efficient DNN Inference on Approximate Accelerators Through Formal Property Exploration

AX-DBN: An Approximate Computing Framework for the Design of Low-Power Discriminative Deep Belief Networks

Training Deep Neural Networks with 8-bit Floating Point Numbers

Energy efficient spiking neural network processing using approximate arithmetic units and variable precision weights

Energy-efficient Neural Networks Using Approximate Computation Reuse.

Effects of Approximation in Computation on the Accuracy and Performance of Deep Neural Network Inference

CoAxNN: Optimizing on-device deep learning with conditional approximate neural networks