Abstract:Gradient descent has been a central training principle for artificial neural networks from the early beginnings to today's deep learning networks. The most common implementation is the backpropagation algorithm for training feed-forward neural networks in a supervised fashion. Backpropagation involves computing the gradient of a loss function, with respect to the weights of the network, to update the weights and thus minimize loss. Although the mean square error is often used as a loss function, the general stochastic gradient descent principle does not immediately connect with a specific loss function. Another drawback of backpropagation has been the search for optimal values of two important training parameters, learning rate and momentum weight, which are determined empirically in most systems. The learning rate specifies the step size towards a minimum of the loss function when following the gradient, while the momentum weight considers previous weight changes when updating current weights. Using both parameters in conjunction with each other is generally accepted as a means to improving training, although their specific values do not follow immediately from standard backpropagation theory. This paper proposes a new information-theoretical loss function motivated by neural signal processing in a synapse. The new loss function implies a specific learning rate and momentum weight, leading to empirical parameters often used in practice. The proposed framework also provides a more formal explanation of the momentum term and its smoothing effect on the training process. All results taken together show that loss, learning rate, and momentum are closely connected. To support these theoretical findings, experiments for handwritten digit recognition show the practical usefulness of the proposed loss function and training parameters.

Convergence of Gradient Method with Momentum for Two-Layer Feedforward Neural Networks

Convergence rates of stochastic gradient method with independent sequences of step-size and momentum weight

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum

On the convergence of gradient descent for two layer neural networks

Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

Continuous Time Analysis of Momentum Methods

Accelerating Federated Learning via Momentum Gradient Descent

A Globally Convergent Gradient Method with Momentum

Provable convergence of Nesterov’s accelerated gradient method for over-parameterized neural networks

Accelerated analysis on the triple momentum method for a two-layer ReLU neural network

Convergence of Gradient Algorithms for Nonconvex C1+α Cost Functions

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

Multi-stage stochastic gradient method with momentum acceleration

Learning-rate-free Momentum SGD with Reshuffling Converges in Nonsmooth Nonconvex Optimization

Steepest Descent with Momentum for Quadratic Functions is a Version of the Conjugate Gradient Method.

The Marginal Value of Momentum for Small Learning Rate SGD

The Golden Ratio of Learning and Momentum

Convergence of the Iterates for Momentum and RMSProp for Local Smooth Functions: Adaptation is the Key

Convergence Analysis of Two-layer Neural Networks with ReLU Activation