Abstract:Stochastic Gradient Descent (SGD) series optimization methods play the vital role in training neural networks, attracting growing attention in science and engineering fields of the intelligent system. The choice of learning rates affects the convergence rate of SGD series optimization methods. Currently, learning rate adjustment strategies mainly face the following problems: (1) The traditional learning rate decay method mainly adopts manual manner during training iterations, the small learning rate produced from which causes slow convergence in training neural networks. (2) Adaptive method (e.g., Adam) has poor generalization performance. To alleviate the above issues, we propose a novel automatic learning rate decay strategy for SGD optimization methods in neural networks. On the basis of the observation that the convergence rate's upper bound enjoys minimization in a specific iteration concerning the current learning rate, we first present the expression of the current learning rate determined by historical learning rates. And merely one extra parameter is initialized to generate automatic decreasing learning rates during the training process. Our proposed approach is applied to SGD and Momentum SGD optimization algorithms, and concrete theoretical proof explains its convergence. Numerical simulations are conducted on the MNIST and Cifar‐10 data sets with different neural networks. Experimental results show that our algorithm outperforms existing classical ones, achieving faster convergence rate, better stability, and generalization performance in neural network training. It also lays a foundation for large‐scale parallel search of initial parameters in intelligent systems.

Gradient Descent Optimization in Deep Learning Model Training Based on Multistage and Method Combination Strategy

RETRACTED: Gradient Descent Optimization in Deep Learning Model Training Based on Multistage and Method Combination Strategy (retracted Article)

Gradient Descent based Optimization Algorithms for Deep Learning Models Training

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

Learning Gradient Descent: Better Generalization and Longer Horizons

Optimal Adaptive and Accelerated Stochastic Gradient Descent

Research on Three-Step Accelerated Gradient Algorithm in Deep Learning

Gradient Descent: The Ultimate Optimizer

Layerwise Optimization by Gradient Decomposition for Continual Learning

Unbiased Quasi-Hyperbolic Nesterov-Gradient Momentum-Based Optimizers for Accelerating Convergence

An automatic learning rate decay strategy for stochastic gradient descent optimization methods in neural networks

Hybrid Quantum-Classical Scheduling for Accelerating Neural Network Training with Newton's Gradient Descent

A compensation-based optimization strategy for top dense layer training

An optimization Strategy for Deep Neural Networks Training

Reconstructing Deep Neural Networks: Unleashing the Optimization Potential of Natural Gradient Descent

Optimization Methods in Deep Learning: A Comprehensive Overview

Optimization for deep learning: theory and algorithms

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario

Convergence of Stochastic Gradient Descent in Deep Neural Network