Abstract:First-order gradient-based optimization algorithms have been of core practical importance in the field of deep learning. In this paper, we propose a new weighting mechanism-based first-order gradient descent optimization algorithm, namely NWM-Adam, to resolve the undesirable convergence behavior of some optimization algorithms which employ fixed sized window of past gradients to scale the gradient updates and improve the performance of Adam and AMSGrad. The NWM-Adam is developed on the basis of the idea, i.e., placing more memory of the past gradients than the recent gradients. Furthermore, it can easily adjust the degree to which how much the past gradients weigh in the estimation. In order to empirically test the performance of our proposed NWM-Adam optimization algorithm, we compare it with other popular optimization algorithms in three well-known machine learning models, i.e., logistic regression, multi-layer fully connected neural networks, and deep convolutional neural networks. The experimental results show that the NWM-Adam can outperform other optimization algorithms.

The Optimization to the Structure of a Continuous Wavelet Neural Network

Genetic Algorithm for Optimization of Neural Network Model of Pipe Network

Self-adaptive GA for Optimization of Neural Network Model of Pipe Network for State Estimation

Wavelet Chaotic Neural Networks and Their Application to Optimization Problems

A Modified Algorithm Based on Wavelet Networks

Novel Algorithm of Wavelet Network Structural Design

A Noval Algorithm for Optimizing Wavelet Networks and Its Application

Continuous wavelet process neural network and its application

Effective Neural Network Training with a New Weighting Mechanism-Based Optimization Algorithm.

Continuous Generative Neural Networks: A Wavelet-Based Architecture in Function Spaces

Tuning Of The Structure And Parmeters Of Wavelet Neural Network Using Improved Chaotic Pso

Exploiting Problem Structure in Deep Declarative Networks: Two Case Studies

Wavelet Neural Networks for Adaptive Equalization

Multi-Scale Non-Linear System Modeling Using Wavelet Networks

Design And Application Of Wavelet Networks Based On Immune Algorithm

Tuning of the Structure and Parameters of Wavelet Neural Network Using Improved Chaotic PSO

Near-optimal Construction of Wavelet Networks for Nonlinear System Modeling

Optimization method for hidden layer nodes of WNN based on rough set reduction

Neural network compression via learnable wavelet transforms

Construction of a New Adaptive Wavelet Network and Its Learning Algorithm

EVOLVING WAVELET NEURAL NETWORKS AND ITS APPLICATION IN CONDITION FORECASTING FOR MACHINE