Abstract:Majorization-minimization schemes are a broad class of iterative methods targeting general optimization problems, including nonconvex, nonsmooth and stochastic. These algorithms minimize successively a sequence of upper bounds of the objective function so that along the iterations the objective value decreases. We present a stochastic higher-order algorithmic framework for minimizing the average of a very large number of sufficiently smooth functions. Our stochastic framework is based on the notion of stochastic higher-order upper bound approximations of the finite-sum objective function and minibatching. We derive convergence results for nonconvex and convex optimization problems when the higher-order approximation of the objective function yields an error that is p times differentiable and has Lipschitz continuous p derivative. More precisely, for general nonconvex problems we present asymptotic stationary point guarantees and under Kurdyka-Lojasiewicz property we derive local convergence rates ranging from sublinear to linear. For convex problems with uniformly convex objective function we derive local (super)linear convergence results for our algorithm. Compared to existing stochastic (first-order) methods, our algorithm adapts to the problem's curvature and allows using any batch size. Preliminary numerical tests support the effectiveness of our algorithmic framework.

Stochastic two-scale convergence in Orlicz-Sobolev's spaces and applications to the homogenization of an integral functional

Homogenization and Two-Scale Convergence

Scaling limits and stochastic homogenization for some nonlinear parabolic equations

On Convergence to Stochastic Integrals

Stochastic-periodic Homogenization of Non-stationary Incompressible Navier-Stokes Type Equations

(Two-scale) $W^{1}L^Φ$-gradient Young measures and homogenization of integral functionals in Orlicz-Sobolev spaces

Reiterated Periodic Homogenization of Parabolic Monotone Operators with Nonstandard Growth

Analysis of Multiscale Methods for Stochastic Differential Equations

Convergence of the family of the deformed Euler–Halley iterations under the Hölder condition of the second derivative

Convergence analysis of stochastic higher-order majorization-minimization algorithms

A stochastic regularized second-order iterative scheme for optimal control and inverse problems in stochastic partial differential equations

$Γ$-convergence and stochastic homogenization of second order singular perturbation models for phase transitions

Constant Step Stochastic Approximations Involving Differential Inclusions: Stability, Long-Run Convergence and Applications

Homogenization principle and numerical analysis for fractional stochastic differential equations with different scales

On Bourgain's approach to stochastic homogenization

On a homogenization problem

New homogenization results for convex integral functionals and their Euler–Lagrange equations

A Simple Collocation-Type Approach to Numerical Stochastic Homogenization

On the Local Convergence of a Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

Existence and convergence results for infinite dimensional nonlinear stochastic equations with multiplicative noise