Abstract:In a real Hilbert space domain setting, we study the convergence properties of the stochastic Ravine accelerated gradient method for convex differentiable optimization. We consider the general form of this algorithm where the extrapolation coefficients can vary with each iteration, and where the evaluation of the gradient is subject to random errors. This general treatment models a breadth of practical algorithms and numerical implementations. We show that, under a proper tuning of the extrapolation parameters, and when the error variance associated with the gradient evaluations or the step-size sequences vanish sufficiently fast, the Ravine method provides fast convergence of the values both in expectation and almost surely. We also improve the convergence rates from O(.) to o(.). Moreover, we show almost sure summability property of the gradients, which implies the fast convergence of the gradients towards zero. This property reflects the fact that the high-resolution ODE of the Ravine method includes a Hessian-driven damping term. When the space is also separable, our analysis allows also to establish almost sure weak convergence of the sequence of iterates provided by the algorithm. We finally specialize the analysis to consider different parameter choices, including vanishing and constant (heavy ball method with friction) damping parameter, and present a comprehensive landscape of the tradeoffs in speed and accuracy associated with these parameter choices and statistical properties on the sequence of errors in the gradient computations. We provide a thorough discussion of the similarities and differences with the Nesterov accelerated gradient which satisfies similar asymptotic convergence rates.

Linear convergence of accelerated conditional gradient algorithms in spaces of measures

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

A Linearly Convergent Conditional Gradient Algorithm with Applications to Online and Stochastic Optimization

Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

Linear Convergence with Condition Number Independent Access of Full Gradients.

Linear Convergence of Adaptive Stochastic Gradient Descent

Linear Convergence of Forward-Backward Accelerated Algorithms without Knowledge of the Modulus of Strong Convexity

Accelerating optimization over the space of probability measures

Convergence Analysis of Accelerated Stochastic Gradient Descent under the Growth Condition

Linear-memory and Decomposition-invariant Linearly Convergent Conditional Gradient Algorithm for Structured Polytopes

Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation

The stochastic Ravine accelerated gradient method with general extrapolation coefficients

Linear Convergence of Stochastic Iterative Greedy Algorithms With Sparse Constraints

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Local Conditions for Global Convergence of Gradient Flows and Proximal Point Sequences in Metric Spaces

Linear Convergence Rates for Extrapolated Fixed Point Algorithms

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

A Methodology Establishing Linear Convergence of Adaptive Gradient Methods under PL Inequality

Gradient Methods with Online Scaling