Abstract:In this work we investigate the practicality of stochastic gradient descent and recently introduced variants with variance-reduction techniques in imaging inverse problems. Such algorithms have been shown in the machine learning literature to have optimal complexities in theory, and provide great improvement empirically over the deterministic gradient methods. Surprisingly, in some tasks such as image deblurring, many of such methods fail to converge faster than the accelerated deterministic gradient methods, even in terms of epoch counts. We investigate this phenomenon and propose a theory-inspired mechanism for the practitioners to efficiently characterize whether it is beneficial for an inverse problem to be solved by stochastic optimization techniques or not. Using standard tools in numerical linear algebra, we derive conditions on the spectral structure of the inverse problem for being a suitable application of stochastic gradient methods. Particularly, we show that, for an imaging inverse problem, if and only if its Hessain matrix has a fast-decaying eigenspectrum, then the stochastic gradient methods can be more advantageous than deterministic methods for solving such a problem. Our results also provide guidance on choosing appropriately the partition minibatch schemes, showing that a good minibatch scheme typically has relatively low correlation within each of the minibatches. Finally, we propose an accelerated primal-dual SGD algorithm in order to tackle another key bottleneck of stochastic optimization which is the heavy computation of proximal operators. The proposed method has fast convergence rate in practice, and is able to efficiently handle non-smooth regularization terms which are coupled with linear operators.

Stochastic Gradient Descent Method with Convex Penalty for Ill-Posed Problems in Banach Spaces

Fast Subspace Optimization Method for Nonlinear Inverse Problems in Banach Spaces with Uniformly Convex Penalty Terms

Iterative Runge-Kutta-Type Methods with Convex Penalty for Inverse Problems in Hilbert Spaces

Stochastic gradient descent for linear inverse problems in variable exponent Lebesgue spaces

Stochastic variance reduced gradient method for linear ill-posed inverse problems

Stochastic Gradient Descent for Linear Inverse Problems in Hilbert Spaces.

On the Convergence of A Data-Driven Regularized Stochastic Gradient Descent for Nonlinear Ill-Posed Problems

The Stochastic Steepest Descent Method for Robust Optimization in Banach Spaces

A Fast Data-Driven Iteratively Regularized Method with Convex Penalty for Solving Ill-Posed Problems

The Practicality of Stochastic Optimization in Imaging Inverse Problems

Stochastic proximal gradient methods for nonconvex problems in Hilbert spaces

Inexact Newton Regularization in Banach Spaces Based on Two-Point Gradient Method with Uniformly Convex Penalty Terms

A Frozen Levenberg-Marquardt-Kaczmarz Method with Convex Penalty Terms and Two-Point Gradient Strategy for Ill-Posed Problems

REGINN-IT Method with General Convex Penalty Terms for Nonlinear Inverse Problems

Levenberg-Marquardt method with general convex penalty for nonlinear inverse problems

An Accelerated Kaczmarz Type Method for Nonlinear Inverse Problems in Banach Spaces with Uniformly Convex Penalty

Stochastic Gradient Descent in the Viewpoint of Graduated Optimization

On the Discrepancy Principle for Stochastic Gradient Descent

An iteration regularization method with general convex penalty for nonlinear inverse problems in Banach spaces

An Alternating Structure-Adapted Bregman Proximal Gradient Descent Algorithm for Constrained Nonconvex Nonsmooth Optimization Problems and Its Inertial Variant.

A Fast Two-Point Gradient Algorithm Based on Sequential Subspace Optimization Method for Nonlinear Ill-Posed Problems