Abstract:In this work we investigate the practicality of stochastic gradient descent and recently introduced variants with variance-reduction techniques in imaging inverse problems. Such algorithms have been shown in the machine learning literature to have optimal complexities in theory, and provide great improvement empirically over the deterministic gradient methods. Surprisingly, in some tasks such as image deblurring, many of such methods fail to converge faster than the accelerated deterministic gradient methods, even in terms of epoch counts. We investigate this phenomenon and propose a theory-inspired mechanism for the practitioners to efficiently characterize whether it is beneficial for an inverse problem to be solved by stochastic optimization techniques or not. Using standard tools in numerical linear algebra, we derive conditions on the spectral structure of the inverse problem for being a suitable application of stochastic gradient methods. Particularly, we show that, for an imaging inverse problem, if and only if its Hessain matrix has a fast-decaying eigenspectrum, then the stochastic gradient methods can be more advantageous than deterministic methods for solving such a problem. Our results also provide guidance on choosing appropriately the partition minibatch schemes, showing that a good minibatch scheme typically has relatively low correlation within each of the minibatches. Finally, we propose an accelerated primal-dual SGD algorithm in order to tackle another key bottleneck of stochastic optimization which is the heavy computation of proximal operators. The proposed method has fast convergence rate in practice, and is able to efficiently handle non-smooth regularization terms which are coupled with linear operators.

Spurious minimizers in non uniform Fourier sampling optimization

Fast Fourier Optimization: Sparsity Matters

The Practicality of Stochastic Optimization in Imaging Inverse Problems

Optimal Compressive Imaging of Fourier Data

SF-SFD: Stochastic Optimization of Fourier Coefficients to Generate Space-Filling Designs

Subsampling at Information Theoretically Optimal Rates

Bayesian Optimization of Sampling Densities in MRI

A Sub-sampled Tensor Method for Non-convex Optimization

Sampling the Fourier transform along radial lines

Optimal Learning Rates for Regularized Least-Squares with a Fourier Capacity Condition

Subsampled Nonmonotone Spectral Gradient Methods

Global Optimization Algorithm through High-Resolution Sampling

Spurious local minima in nonconvex sum-of-squares optimization

A Gradient Smoothed Functional Algorithm with Truncated Cauchy Random Perturbations for Stochastic Optimization

Optimal sampling for stochastic and natural gradient descent

Frequency Extrapolation by Nonconvex Compressive Sensing

With or Without Replacement? Improving Confidence in Fourier Imaging

A Decoupled Approach for Composite Sparse-plus-Smooth Penalized Optimization

Signal reconstruction using determinantal sampling

Joint Sampling and Optimisation for Inverse Rendering

Sampling numbers of smoothness classes via $\ell^1$-minimization