Abstract:Gradient-based (a.k.a. `first order') optimization algorithms are routinely used to solve large scale non-convex problems. Yet, it is generally hard to predict their effectiveness. In order to gain insight into this question, we revisit the problem of solving $n$ random equations in $d$ variables. We assume that the equations are independent realizations of a common Gaussian process. A special case is the one of random polynomials, which has been studied since Littlewood-Offord and Kac in the 1940s, and Shub-Smale in the 1990s. The last authors first investigated the computational aspect of this problem. Smale's `17th problem' asks whether a system of random polynomial equations can be (approximately) solved in average-case polynomial time. We formulate this as a nonconvex optimization problem, and develop gradient and Hessian-based algorithms to solve it. Leveraging recent advances in spin glass theory, we characterize the optimal algorithm in this class, and show that it undergoes a phase transition when $\alpha=n/d$ crosses a threshold. For $\alpha>\alpha_{\text{alg}}$ solutions may exist (depending on the distribution of the equations) but are not found by local algorithms. We compare these predictions with numerical experiments and observe that stochastic gradient descent approaches the optimal algorithm. We show that the geometry of solutions in a neighborhood of a random initialization undergoes a phase transition when $\alpha$ crosses a threshold $\alpha_{\text{sens}}$ (for `sensitivity') smaller than $\alpha_{\text{alg}}$. This geometric phase transition has algorithmic implications. Finally, we observe that the dynamics of these algorithms exhibits remarkable universality with respect to the details of the cost function.

Random Linear Systems with Quadratic Constraints: from Random Matrix Theory to replicas and back

Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems

Randomized Methods for Linear Constraints: Convergence Rates and Conditioning

Randomized Iterative Methods for Linear Systems

Randomized methods for rank-deficient linear systems

A semi-algebraic view on quadratic constraints for polynomial systems

On some features of Quadratic Unconstrained Binary Optimization with random coefficients

Least squares solutions of bilinear equations

Solving systems of Random Equations via First and Second-Order Optimization Algorithms

Linear-Quadratic Problems in Systems and Controls via Covariance Representations and Linear-Conic Duality: Finite-Horizon Case

A Stochastic Sequential Quadratic Optimization Algorithm for Nonlinear Equality Constrained Optimization with Rank-Deficient Jacobians

Randomised subspace methods for non-convex optimization, with applications to nonlinear least-squares

An approximation theory of matrix rank minimization and its application to quadratic equations

A Matrix-Free Trust-Region SQP Method for Equality Constrained Optimization

On the Least Squares Solutions of a System of Bilinear Equations

Solving Dense Linear Systems Faster Than via Preconditioning

Large-Scale Optimization with Linear Equality Constraints Using Reduced Compact Representation

A framework for randomized time-splitting in linear-quadratic optimal control

Surrounding the solution of a linear system of equations from all sides

Linear Phase Transition in Random Linear Constraint Satisfaction Problem

Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization