Abstract:Gradient descent has been a central training principle for artificial neural networks from the early beginnings to today's deep learning networks. The most common implementation is the backpropagation algorithm for training feed-forward neural networks in a supervised fashion. Backpropagation involves computing the gradient of a loss function, with respect to the weights of the network, to update the weights and thus minimize loss. Although the mean square error is often used as a loss function, the general stochastic gradient descent principle does not immediately connect with a specific loss function. Another drawback of backpropagation has been the search for optimal values of two important training parameters, learning rate and momentum weight, which are determined empirically in most systems. The learning rate specifies the step size towards a minimum of the loss function when following the gradient, while the momentum weight considers previous weight changes when updating current weights. Using both parameters in conjunction with each other is generally accepted as a means to improving training, although their specific values do not follow immediately from standard backpropagation theory. This paper proposes a new information-theoretical loss function motivated by neural signal processing in a synapse. The new loss function implies a specific learning rate and momentum weight, leading to empirical parameters often used in practice. The proposed framework also provides a more formal explanation of the momentum term and its smoothing effect on the training process. All results taken together show that loss, learning rate, and momentum are closely connected. To support these theoretical findings, experiments for handwritten digit recognition show the practical usefulness of the proposed loss function and training parameters.

How Does Learning Rate Decay Help Modern Neural Networks?

Learning Stages: Phenomenon, Root Cause, Mechanism Hypothesis, and Implications.

Learning Stages: Phenomenon, Root Cause, Mechanism Hypothesis, and Implications

An automatic learning rate decay strategy for stochastic gradient descent optimization methods in neural networks

Understanding the Generalization Benefits of Late Learning Rate Decay

Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate

Adaptive Weight Decay for Deep Neural Networks

Where Do Large Learning Rates Lead Us?

Why Do We Need Weight Decay in Modern Deep Learning?

Exponential decay sine wave learning rate for fast deep neural network training

Learning Rate Dropout

On Learning Rates and Schrödinger Operators

Advancing neural network calibration: The role of gradient decay in large-margin Softmax optimization

An optimization Strategy for Deep Neural Networks Training

An Adaptive Mechanism to Achieve Learning Rate Dynamically

The Golden Ratio of Learning and Momentum

Optimal Linear Decay Learning Rate Schedules and Further Refinements

LLR: Learning Learning Rates by LSTM for Training Neural Networks.

Super Level Sets and Exponential Decay: A Synergistic Approach to Stable Neural Network Training

Large Learning Rates Improve Generalization: But How Large Are We Talking About?

How to decay your learning rate