Abstract:In this work we investigate the practicality of stochastic gradient descent and recently introduced variants with variance-reduction techniques in imaging inverse problems. Such algorithms have been shown in the machine learning literature to have optimal complexities in theory, and provide great improvement empirically over the deterministic gradient methods. Surprisingly, in some tasks such as image deblurring, many of such methods fail to converge faster than the accelerated deterministic gradient methods, even in terms of epoch counts. We investigate this phenomenon and propose a theory-inspired mechanism for the practitioners to efficiently characterize whether it is beneficial for an inverse problem to be solved by stochastic optimization techniques or not. Using standard tools in numerical linear algebra, we derive conditions on the spectral structure of the inverse problem for being a suitable application of stochastic gradient methods. Particularly, we show that, for an imaging inverse problem, if and only if its Hessain matrix has a fast-decaying eigenspectrum, then the stochastic gradient methods can be more advantageous than deterministic methods for solving such a problem. Our results also provide guidance on choosing appropriately the partition minibatch schemes, showing that a good minibatch scheme typically has relatively low correlation within each of the minibatches. Finally, we propose an accelerated primal-dual SGD algorithm in order to tackle another key bottleneck of stochastic optimization which is the heavy computation of proximal operators. The proposed method has fast convergence rate in practice, and is able to efficiently handle non-smooth regularization terms which are coupled with linear operators.

Stochastic Inverse Problem: stability, regularization and Wasserstein gradient flow

Differential-Equation Constrained Optimization With Stochasticity

Differential Equation–Constrained Optimization with Stochasticity

Statistical inverse learning problems with random observations

The Practicality of Stochastic Optimization in Imaging Inverse Problems

Derivative-free stochastic bilevel optimization for inverse problems

On log-concave approximations of high-dimensional posterior measures and stability properties in non-linear inverse problems

Inverse problems for stochastic partial differential equations

Sparse identification of nonlocal interaction kernels in nonlinear gradient flow equations via partial inversion

Stein Variational Gradient Descent on Infinite-Dimensional Space and Applications to Statistical Inverse Problems

Inverse Problems for Stochastic Partial Differential Equations: Some Progresses and Open Problems

Wasserstein Gradient Flow over Variational Parameter Space for Variational Inference

Statistical Learning and Inverse Problems: A Stochastic Gradient Approach

Inverse problems for coupled nonlocal nonlinear systems arising in mathematical biology

A stochastic regularized second-order iterative scheme for optimal control and inverse problems in stochastic partial differential equations

An inverse Cauchy problem of a stochastic hyperbolic equation

Solving Fredholm Integral Equations of the First Kind via Wasserstein Gradient Flows

Stochastic Data-Driven Bouligand Landweber Method for Solving Non-smooth Inverse Problems

Continuous-time Riemannian SGD and SVRG Flows on Wasserstein Probabilistic Space

Wasserstein-based Projections with Applications to Inverse Problems

Large-Scale Wasserstein Gradient Flows