Global convergence of gradient descent for phase retrieval

Théodore Fougereux,Cédric Josz,Xiaopeng Li

2024-10-17

Abstract:We propose a tensor-based criterion for benign landscape in phase retrieval and establish boundedness of gradient trajectories. This implies that gradient descent will converge to a global minimum for almost every initial point.

Optimization and Control,Machine Learning

What problem does this paper attempt to address?

On Global Convergence of Gradient Descent Algorithms for Generalized Phase Retrieval Problem.

Li Ji,Zhou Tie

DOI: https://doi.org/10.1016/j.cam.2017.07.008

IF: 2.872

2018-01-01

Journal of Computational and Applied Mathematics

Abstract:In this paper, we study the generalized phase retrieval problem: to recover a signal x∈Cn from the measurements yr=|〈ar,x〉|2, r=1,2,…,m. The problem can be reformulated as a least-squares minimization problem. Although the cost function is nonconvex, the global convergence of gradient descent algorithms from a random initialization is studied, when m is large enough. We improve the known result of the local convergence from a spectral initialization. When the signal x is real-valued, we prove that the cost function is local convex near the solution {±x}. To accelerate the gradient descent, we review and apply several efficient line search methods with exact line search stepsize. We also perform a comparative numerical study of the line search methods and the alternative projection method. Numerical simulations demonstrate the superior ability of LBFGS algorithm than other algorithms.
Nearly optimal bounds for the global geometric landscape of phase retrieval

Jian-Feng Cai,Meng Huang,Dong Li,Yang Wang

DOI: https://doi.org/10.1088/1361-6420/acdab7

IF: 2.408

2023-06-02

Inverse Problems

Abstract:The phase retrieval problem is concerned with recovering an unknown signal from a set of magnitude-only measurements . A natural least squares formulation can be used to solve this problem efficiently even with random initialization, despite its non-convexity of the loss function. One way to explain this surprising phenomenon is the benign geometric landscape: (1) all local minimizers are global; and (2) the objective function has a negative curvature around each saddle point and local maximizer. In this paper, we show that Gaussian random measurements are sufficient to guarantee the loss function of a commonly used estimator has such benign geometric landscape with high probability. This is a step toward answering the open problem given by Sun-Qu-Wright \cite{Sun18}, in which the authors suggest that or even is enough to guarantee the favorable geometric property.

mathematics, applied,physics, mathematical
Convergence Analysis of Projected Gradient Descent for Schatten-p Nonconvex Matrix Recovery

Yun Cai,Song Li

DOI: https://doi.org/10.1007/s11425-014-4949-1

2014-01-01

Science China Mathematics

Abstract:The matrix rank minimization problem arises in many engineering applications. As this problem is NP-hard, a nonconvex relaxation of matrix rank minimization, called the Schatten-p quasi-norm minimization (0 < p < 1), has been developed to approximate the rank function closely. We study the performance of projected gradient descent algorithm for solving the Schatten-p quasi-norm minimization (0 < p < 1) problem. Based on the matrix restricted isometry property (M-RIP), we give the convergence guarantee and error bound for this algorithm and show that the algorithm is robust to noise with an exponential convergence rate.
Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Hang Xu,Song Li,Junhong Lin

DOI: https://doi.org/10.1007/s10444-024-10163-2

2024-01-01

Advances in Computational Mathematics

Abstract:Many problems in data science can be treated as recovering structural signals from a set of linear measurements, sometimes perturbed by dense noise or sparse corruptions. In this paper, we develop a unified framework of considering a nonsmooth formulation with sparse or low-rank constraint for meeting the challenges of mixed noises—bounded noise and sparse noise. We show that the nonsmooth formulations of the problems can be well solved by the projected subgradient methods at a rapid rate when initialized at any points. Consequently, nonsmooth loss functions ( ℓ _1 -minimization programs) are naturally robust against sparse noise. Our framework simplifies and generalizes the existing analyses including compressed sensing, matrix sensing, quadratic sensing, and bilinear sensing. Motivated by recent work on the stochastic gradient method, we also give some experimentally and theoretically preliminary results about the projected stochastic subgradient method.
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Francesca Mignacco,Pierfrancesco Urbani,Lenka Zdeborová

DOI: https://doi.org/10.1088/2632-2153/ac0615

2021-07-13

Machine Learning: Science and Technology

Abstract:Abstract In this paper we investigate how gradient-based algorithms such as gradient descent (GD), (multi-pass) stochastic GD, its persistent variant, and the Langevin algorithm navigate non-convex loss-landscapes and which of them is able to reach the best generalization error at limited sample complexity. We consider the loss landscape of the high-dimensional phase retrieval problem as a prototypical highly non-convex example. We observe that for phase retrieval the stochastic variants of GD are able to reach perfect generalization for regions of control parameters where the GD algorithm is not. We apply dynamical mean-field theory from statistical physics to characterize analytically the full trajectories of these algorithms in their continuous-time limit, with a warm start, and for large system sizes. We further unveil several intriguing properties of the landscape and the algorithms such as that the GD can obtain better generalization properties from less informed initializations.
Convolutional Phase Retrieval via Gradient Descent

Qing Qu,Yuqian Zhang,Yonina C. Eldar,John Wright

DOI: https://doi.org/10.48550/arXiv.1712.00716

2019-10-06

Abstract:We study the convolutional phase retrieval problem, of recovering an unknown signal $\mathbf x \in \mathbb C^n $ from $m$ measurements consisting of the magnitude of its cyclic convolution with a given kernel $\mathbf a \in \mathbb C^m $. This model is motivated by applications such as channel estimation, optics, and underwater acoustic communication, where the signal of interest is acted on by a given channel/filter, and phase information is difficult or impossible to acquire. We show that when $\mathbf a$ is random and the number of observations $m$ is sufficiently large, with high probability $\mathbf x$ can be efficiently recovered up to a global phase shift using a combination of spectral initialization and generalized gradient descent. The main challenge is coping with dependencies in the measurement operator. We overcome this challenge by using ideas from decoupling theory, suprema of chaos processes and the restricted isometry property of random circulant matrices, and recent analysis of alternating minimization methods.

Computation,Information Theory,Numerical Analysis,Optimization and Control,Machine Learning
Globally Convergent Levenberg-Marquardt Method for Phase Retrieval.

Chao Ma,Xin Liu,Zaiwen Wen

DOI: https://doi.org/10.1109/tit.2018.2881187

IF: 2.5

2019-01-01

IEEE Transactions on Information Theory

Abstract:In this paper, we consider a nonlinear least squares model for the phase retrieval problem. Since the Hessian matrix may not be positive definite and the Gauss-Newton (GN) matrix is singular at any optimal solution, we propose a modified Levenberg-Marquardt (LM) method, where the Hessian is substituted by a summation of the GN matrix and a regularization term. Similar to the well-known Wirtinger flow (WF) algorithm under certain assumptions, we start from an initial point provably close to the set of the global optimal solutions. Global linear convergence and local quadratic convergence to the global solution set are established by estimating the smallest nonzero eigenvalues of the GN matrix, establishing local error bound properties and constructing a modified regularization condition. The computational cost becomes tractable if a preconditioned conjugate gradient method is applied to solve the LM equation inexactly. Specifically, the pre-conditioner is constructed from the expectation of the LM coefficient matrix by assuming the independence between the measurements and iteration point. The initialization can be dropped by slightly modifying the algorithm. Preliminary numerical experiments show that our algorithm is robust and it is often faster than the WF method on both random examples and natural image recovery.
The Local Landscape of Phase Retrieval Under Limited Samples

Kaizhao Liu,Zihao Wang,Lei Wu

2024-10-12

Abstract:In this paper, we present a fine-grained analysis of the local landscape of phase retrieval under the regime of limited samples. Specifically, we aim to ascertain the minimal sample size required to guarantee a benign local landscape surrounding global minima in high dimensions. Let $n$ and $d$ denote the sample size and input dimension, respectively. We first explore the local convexity and establish that when $n=o(d\log d)$, for almost every fixed point in the local ball, the Hessian matrix has negative eigenvalues, provided $d$ is sufficiently large. % Consequently, the local landscape is highly non-convex. We next consider the one-point convexity and show that, as long as $n=\omega(d)$, with high probability, the landscape is one-point strongly convex in the local annulus: $\{w\in\mathbb{R}^d: o_d(1)\leqslant \|w-w^*\|\leqslant c\}$, where $w^*$ is the ground truth and $c$ is an absolute constant. This implies that gradient descent, initialized from any point in this domain, can converge to an $o_d(1)$-loss solution exponentially fast. Furthermore, we show that when $n=o(d\log d)$, there is a radius of $\widetilde\Theta\left(\sqrt{1/d}\right)$ such that one-point convexity breaks down in the corresponding smaller local ball. This indicates an impossibility of establishing a convergence to the exact $w^*$ for gradient descent under limited samples by relying solely on one-point convexity.

Information Theory,Machine Learning,Signal Processing,Optimization and Control,Statistics Theory
Robust Gradient Descent for Phase Retrieval

Alex Buna,Patrick Rebeschini

2024-10-14

Abstract:Recent progress in robust statistical learning has mainly tackled convex problems, like mean estimation or linear regression, with non-convex challenges receiving less attention. Phase retrieval exemplifies such a non-convex problem, requiring the recovery of a signal from only the magnitudes of its linear measurements, without phase (sign) information. While several non-convex methods, especially those involving the Wirtinger Flow algorithm, have been proposed for noiseless or mild noise settings, developing solutions for heavy-tailed noise and adversarial corruption remains an open challenge. In this paper, we investigate an approach that leverages robust gradient descent techniques to improve the Wirtinger Flow algorithm's ability to simultaneously cope with fourth moment bounded noise and adversarial contamination in both the inputs (covariates) and outputs (responses). We address two scenarios: known zero-mean noise and completely unknown noise. For the latter, we propose a preprocessing step that alters the problem into a new format that does not fit traditional phase retrieval approaches but can still be resolved with a tailored version of the algorithm for the zero-mean noise context.

Machine Learning
Asymptotic quadratic convergence of the Gauss-Newton method for complex phase retrieval

Meng Huang

2024-06-14

Abstract:In this paper, we introduce a Gauss-Newton method for solving the complex phase retrieval problem. In contrast to the real-valued setting, the Gauss-Newton matrix for complex-valued signals is rank-deficient and, thus, non-invertible. To address this, we utilize a Gauss-Newton step that moves orthogonally to certain trivial directions. We establish that this modified Gauss-Newton step has a closed-form solution, which corresponds precisely to the minimal-norm solution of the associated least squares problem. Additionally, using the leave-one-out technique, we demonstrate that $m\ge O( n\log^3 n)$ independent complex Gaussian random measurements ensures that the entire trajectory of the Gauss-Newton iterations remains confined within a specific region of incoherence and contraction with high probability. This finding allows us to establish the asymptotic quadratic convergence rate of the Gauss-Newton method without the need of sample splitting.

Numerical Analysis,Information Theory
Conjugate Gradient Method for Phase Retrieval Based on the Wirtinger Derivative

Zhun Wei,Wen Chen,Cheng-Wei Qiu,Xudong Chen

DOI: https://doi.org/10.1364/josaa.34.000708

2017-01-01

Abstract:A conjugate gradient Wirtinger flow (CG-WF) algorithm for phase retrieval is proposed in this paper. It is shown that, compared with recently reported Wirtinger flow and its modified methods, the proposed CG-WF algorithm is able to dramatically accelerate the convergence rate while keeping the dominant computational cost of each iteration unchanged. We numerically illustrate the effectiveness of our method in recovering 1D Gaussian signals and 2D natural color images under both Gaussian and coded diffraction pattern models.
Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Jinhua Wang,Xiangmei Wang,Chong Li,Jen-Chih Yao

DOI: https://doi.org/10.1137/19m1289285

IF: 2.763

2021-01-01

SIAM Journal on Optimization

Abstract:We study the convergence issue for the gradient algorithm (employing general step sizes) for optimization problems on general Riemannian manifolds (without curvature constraints). Under the assumption of the local convexity/quasi-convexity (resp. weak sharp minima), local/global convergence (resp. linear convergence) results are established. As an application, the linear convergence properties of the gradient algorithm employing the constant step sizes and the Armijo step sizes for finding the Riemannian $L^p$ ($p\in[1,+\infty)$) centers of mass are explored, respectively, which in particular extend and/or improve the corresponding results in \cite{Afsari2013}.
Stable Phase Retrieval with Mirror Descent

Jean-Jacques Godeme,Jalal Fadili,Claude Amra,Myriam Zerrad

2024-06-20

Abstract:In this paper, we aim to reconstruct an n-dimensional real vector from m phaseless measurements corrupted by an additive noise. We extend the noiseless framework developed in [15], based on mirror descent (or Bregman gradient descent), to deal with noisy measurements and prove that the procedure is stable to (small enough) additive noise. In the deterministic case, we show that mirror descent converges to a critical point of the phase retrieval problem, and if the algorithm is well initialized and the noise is small enough, the critical point is near the true vector up to a global sign change. When the measurements are i.i.d Gaussian and the signal-to-noise ratio is large enough, we provide global convergence guarantees that ensure that with high probability, mirror descent converges to a global minimizer near the true vector (up to a global sign change), as soon as the number of measurements m is large enough. The sample complexity bound can be improved if a spectral method is used to provide a good initial guess. We complement our theoretical study with several numerical results showing that mirror descent is both a computationally and statistically efficient scheme to solve the phase retrieval problem.

Optimization and Control,Computer Vision and Pattern Recognition,Information Theory
From Zero to Hero: How local curvature at artless initial conditions leads away from bad minima

Tony Bonnaire,Giulio Biroli,Chiara Cammarota

2024-09-23

Abstract:We provide an analytical study of the evolution of the Hessian during gradient descent dynamics, and relate a transition in its spectral properties to the ability of finding good minima. We focus on the phase retrieval problem as a case study for complex loss landscapes. We first characterize the high-dimensional limit where both the number $M$ and the dimension $N$ of the data are going to infinity at fixed signal-to-noise ratio $\alpha = M/N$. For small $\alpha$, the Hessian is uninformative with respect to the signal. For $\alpha$ larger than a critical value, the Hessian displays at short-times a downward direction pointing towards good minima. While descending, a transition in the spectrum takes place: the direction is lost and the system gets trapped in bad minima. Hence, the local landscape is benign and informative at first, before gradient descent brings the system into a uninformative maze. Through both theoretical analysis and numerical experiments, we show that this dynamical transition plays a crucial role for finite (even very large) $N$: it allows the system to recover the signal well before the algorithmic threshold corresponding to the $N\rightarrow\infty$ limit. Our analysis sheds light on this new mechanism that facilitates gradient descent dynamics in finite dimensions, and highlights the importance of a good initialization based on spectral properties for optimization in complex high-dimensional landscapes.

Machine Learning,Disordered Systems and Neural Networks,Statistical Mechanics
Lévy Gradient Descent: Augmented Random Search for Geophysical Inverse Problems

Zhang Yi,Xu Yixian,Yang Bo

DOI: https://doi.org/10.1007/s10712-021-09644-6

2021-01-01

Surveys in Geophysics

Abstract:A new efficient random search algorithm is introduced for solving inversion problems in geophysical studies. The proposed algorithm is inherently a stochastic optimization method which is built on the concept of gradient descending and Lévy flights. Therefore, the algorithm is referred to as the Lévy gradient descent (L-GD). In which, the Lévy flights is a special class of random walk which consists of many short steps along with a few large steps. Such movements are observed in a varying range of fields, including animals’ foraging patterns, fluid dynamics, transport of light and so on. Meanwhile, the Lévy flights typically shows much higher speed in searching for sparsely located targets compared to the well-known Brownian walks, which make them preferable to drive random search algorithms. As shown in the paper, besides optimal solutions of the inverse problems, the L-GD algorithm could also produce estimations on the error distributions of the resultant model parameters. Following a detailed introduction of the methodology, parameter settings of the algorithm are discussed in length through statistical experiments. Subsequently, the proposed algorithm is evaluated using numeric tests and shows attracting properties of the global convergence and significant higher searching efficiency compared to commonly adopted stochastic optimization techniques in geophysical inversions. Moreover, the L-GD algorithm is applied to the inversions of gravity and seismic travel time data and has achieved the same accuracy as gradient-based optimization methods. Meanwhile, though error estimations generated by L-GD algorithm are essentially qualitative, they could still provide valuable information to help evaluating the resultant model parameters, which is of great importance for practical geophysical inversions.
Riemannian Optimization for Phase Retrieval from Masked Fourier Measurements

Li, Huiping,Li, Song

DOI: https://doi.org/10.1007/s10444-021-09914-2

2021-01-01

Advances in Computational Mathematics

Abstract:In this paper, we consider the noisy phase retrieval problem under the measurements of Fourier transforms with complex random masks. Here two kinds of Riemannian optimization algorithms, namely, Riemannian gradient descent algorithm (RGrad) and Riemannian conjugate gradient descent algorithm (RCG), are presented to solve such problem from these special but widely used measurements in practical applications. Since the masked Fourier measurements are less random, we establish stable guarantees for signals by the truncated variants of RGrad and RCG, respectively. First of all, a good initialization is constructed by means of a truncated spectral method. Then we prove that a signal x∈ℂ^n can be recovered robustly to bounded noise through these two algorithms, provided that L= O(log n) complex random masks are performed in the measurement process. This implies that the sample complexity is optimal up to a log factor, namely, O(nlog n) . Particularly, each sequence generated by the truncated RGrad and RCG provably converges to the true solution at a geometric rate in the noiseless case. Finally, several empirical experiments are provided to show the effectiveness and stability of such two kinds of algorithms compared with Wirtinger Flow(WF) algorithm, for which provable guarantee has been set up for signals under masked Fourier measurements, provided that L= O(log ^4n) .
Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimation

Dejiao Zhang,Laura Balzano

DOI: https://doi.org/10.48550/arXiv.1506.07405

2016-06-25

Abstract:It has been observed in a variety of contexts that gradient descent methods have great success in solving low-rank matrix factorization problems, despite the relevant problem formulation being non-convex. We tackle a particular instance of this scenario, where we seek the $d$-dimensional subspace spanned by a streaming data matrix. We apply the natural first order incremental gradient descent method, constraining the gradient method to the Grassmannian. In this paper, we propose an adaptive step size scheme that is greedy for the noiseless case, that maximizes the improvement of our metric of convergence at each data index $t$, and yields an expected improvement for the noisy case. We show that, with noise-free data, this method converges from any random initialization to the global minimum of the problem. For noisy data, we provide the expected convergence rate of the proposed algorithm per iteration.

Numerical Analysis,Machine Learning
Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

Cong Ma,Kaizheng Wang,Yuejie Chi,Yuxin Chen

DOI: https://doi.org/10.1007/s10208-019-09429-9

2019-07-30

Abstract:Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order to guarantee fast convergence. For vanilla procedures such as gradient descent, however, prior theory either recommends highly conservative learning rates to avoid overshooting, or completely lacks performance guarantees. This paper uncovers a striking phenomenon in nonconvex optimization: even in the absence of explicit regularization, gradient descent enforces proper regularization implicitly under various statistical models. In fact, gradient descent follows a trajectory staying within a basin that enjoys nice geometry, consisting of points incoherent with the sampling mechanism. This "implicit regularization" feature allows gradient descent to proceed in a far more aggressive fashion without overshooting, which in turn results in substantial computational savings. Focusing on three fundamental statistical estimation problems, i.e. phase retrieval, low-rank matrix completion, and blind deconvolution, we establish that gradient descent achieves near-optimal statistical and computational guarantees without explicit regularization. In particular, by marrying statistical modeling with generic optimization theory, we develop a general recipe for analyzing the trajectories of iterative algorithms via a leave-one-out perturbation argument. As a byproduct, for noisy matrix completion, we demonstrate that gradient descent achieves near-optimal error control --- measured entrywise and by the spectral norm --- which might be of independent interest.

Machine Learning,Information Theory,Optimization and Control,Statistics Theory
Global convergence of the gradient method for functions definable in o-minimal structures

Cédric Josz

DOI: https://doi.org/10.1007/s10107-023-01937-5

2024-11-29

Abstract:We consider the gradient method with variable step size for minimizing functions that are definable in o-minimal structures on the real field and differentiable with locally Lipschitz gradients. We prove that global convergence holds if continuous gradient trajectories are bounded, with the minimum gradient norm vanishing at the rate $o(1/k)$ if the step sizes are greater than a positive constant. If additionally the gradient is continuously differentiable, all saddle points are strict, and the step sizes are constant, then convergence to a local minimum holds almost surely over any bounded set of initial points.

Optimization and Control
Strong Convexity of Affine Phase Retrieval

Meng Huang,Zhiqiang Xu

DOI: https://doi.org/10.1109/tsp.2024.3371170

IF: 4.875

2024-03-15

IEEE Transactions on Signal Processing

Abstract:Affine phase retrieval refers to the process of recovering a signal from intensity measurements, with some entries known in advance. In this paper, we demonstrate that a natural least squares formulation for affine phase retrieval is strongly convex on the complete space under certain mild conditions, given that the measurement vectors are complex Gaussian random vectors and that the number of measurements is , where is the dimension of the signals. Based on the result, we prove that the simple gradient descent method for the affine phase retrieval converges linearly to the target solution with high probability from an arbitrary initial point. These results highlight a fundamental difference between affine phase retrieval and classical phase retrieval, where the least squares formulations for classical phase retrieval are non-convex.

engineering, electrical & electronic

Global convergence of gradient descent for phase retrieval

On Global Convergence of Gradient Descent Algorithms for Generalized Phase Retrieval Problem.

Nearly optimal bounds for the global geometric landscape of phase retrieval

Convergence Analysis of Projected Gradient Descent for Schatten-p Nonconvex Matrix Recovery

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Convolutional Phase Retrieval via Gradient Descent

Globally Convergent Levenberg-Marquardt Method for Phase Retrieval.

The Local Landscape of Phase Retrieval Under Limited Samples

Robust Gradient Descent for Phase Retrieval

Asymptotic quadratic convergence of the Gauss-Newton method for complex phase retrieval

Conjugate Gradient Method for Phase Retrieval Based on the Wirtinger Derivative

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Stable Phase Retrieval with Mirror Descent

From Zero to Hero: How local curvature at artless initial conditions leads away from bad minima

Lévy Gradient Descent: Augmented Random Search for Geophysical Inverse Problems

Riemannian Optimization for Phase Retrieval from Masked Fourier Measurements

Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimation

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

Global convergence of the gradient method for functions definable in o-minimal structures

Strong Convexity of Affine Phase Retrieval