Abstract:Gradient descent is the core and foundation of neural networks, and gradient descent optimization heuristics have greatly accelerated progress in deep learning. Although these methods are simple and effective, how they work remains unknown. Gradient descent optimization in deep learning has become a hot research topic. Some research efforts have tried to combine multiple methods to assist network training, but these methods seem to be more empirical, without theoretical guides. In this paper, a framework is proposed to illustrate the principle of combining different gradient descent optimization methods by analyzing several adaptive methods and other learning rate methods. Furthermore, inspired by the principle of warmup, CLR, and SGDR, the concept of multistage is introduced into the field of gradient descent optimization, and a gradient descent optimization strategy in deep learning model training based on multistage and method combination strategy is presented. The effectiveness of the proposed strategy is verified on the massive deep learning network training experiments.

RETRACTED ARTICLE: Training an Innovative Physics-Learned Deep Neural Network by the Adam Optimization Method to Simulate the Responses of the Hybrid Multilayer System

Retracted: Design of Financial Risk Control Model Based on Deep Learning Neural Network

Retracted: Training a Feedforward Neural Network Using Hybrid Gravitational Search Algorithm with Dynamic Multiswarm Particle Swarm Optimization

Retracted: Load Prediction Model of Athletes' Physical Training Competition Based on Nonlinear Algorithm Combined with Ultrasound

Retraction: Yuan, N., et al. Laplacian Eigenmaps Feature Conversion and Particle Swarm Optimization-Based Deep Neural Network for Machine Condition Monitoring. Appl. Sci. 2018, 8, 2611, doi:10.3390/app8122611

Retracted: Motion Trajectory Error of Robotic Arm Based on Neural Network Algorithm

Retracted: Video Tactical Intelligence Analysis Method of Karate Competition Based on Convolutional Neural Network

Retracted: Deep Learning-Guided Simulated Annealing for Designing Vocational High Educational System

Retracted: Development of Basketball Tactics Basic Cooperation Teaching System Based on CNN and BP Neural Network

RETRACTION: The Research of Chinese Martial Arts Cross-Media Communication System Based on Deep Neural Network

Retracted: Swimming Training Evaluation Method Based on Convolutional Neural Network

Retracted: Financial Market Sentiment Prediction Technology and Application Based on Deep Learning Model

Retracted: Intelligent Evaluation of Product Form Design Based on Deep Neural Network Model

RETRACTED: A Modular Neural Network Architecture with Concept

Retracted: Design and Research of the AI Badminton Model Based on the Deep Learning Neural Network

Retracted: Sports Training Strategy and Interactive Control Method Based on Digital Twins and Wireless Sensor Networks

Retracted: Application of the PBL Model Based on Deep Learning in Physical Education Classroom Integrating Production and Education

Retracted: Adoption of Convolutional Neural Network Algorithm Combined with Augmented Reality in Building Data Visualization and Intelligent Detection

RETRACTED: Gradient Descent Optimization in Deep Learning Model Training Based on Multistage and Method Combination Strategy (retracted Article)

WITHDRAWN: High-fidelity, generalizable light-field reconstruction of biological dynamics with physics-informed meta neural representation

Retracted: Detection Method of Limb Movement in Competitive Sports Training Based on Deep Learning