Abstract:We propose a new randomized method for solving systems of nonlinear equations, which can find sparse solutions or solutions under certain simple constraints. The scheme only takes gradients of component functions and uses Bregman projections onto the solution space of a Newton equation. In the special case of euclidean projections, the method is known as nonlinear Kaczmarz method. Furthermore, if the component functions are nonnegative, we are in the setting of optimization under the interpolation assumption and the method reduces to SGD with the recently proposed stochastic Polyak step size. For general Bregman projections, our method is a stochastic mirror descent with a novel adaptive step size. We prove that in the convex setting each iteration of our method results in a smaller Bregman distance to exact solutions as compared to the standard Polyak step. Our generalization to Bregman projections comes with the price that a convex one-dimensional optimization problem needs to be solved in each iteration. This can typically be done with globalized Newton iterations. Convergence is proved in two classical settings of nonlinearity: for convex nonnegative functions and locally for functions which fulfill the tangential cone condition. Finally, we show examples in which the proposed method outperforms similar methods with the same memory requirements.

A Modified Newton Method for Multilinear PageRank

Convergence of Newton'S Method for Systems of Equations with Constant Rank Derivatives

Convergence Criterion of Newton's Method for Singular Systems with Constant Rank Derivatives

Newton's Method for Sections on Riemannian Manifolds: Generalized Covariant Α-Theory

Newton's Method for Underdetermined Systems of Equations under the Γ-Condition

Newton's Method on Riemannian Manifolds: Smale's Point Estimate Theory under the Γ-Condition

Weak dangling block reordering and multi-step block compression for efficiently computing and updating PageRank solutions

Newton's method on two step nilpotent lie groups: convergence criteria and the extended smale point estimate theory

Nonlinear Modified PageRank Problem for Local Graph Partitioning

Newton-type Methods for Inverse Singular Value Problems with Multiple Singular Values

A modification of Newton's method solving non-linear equations with singular Jacobian

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

A Regularized Newton Method for Monotone Nonlinear Equations and Its Application

A structured modified Newton approach for solving systems of nonlinear equations arising in interior-point methods for quadratic programming

A modified Newton method for solving non-symmetric algebraic Riccati equations arising in transport theory

Anderson Accelerated Fixed-Point Iteration for Multilinear PageRank

Multilinear PageRank: Uniqueness, Error Bound and Perturbation Analysis

A Riemannian Proximal Newton Method

Online Learning Guided Curvature Approximation: A Quasi-Newton Method with Global Non-Asymptotic Superlinear Convergence

A Bregman-Kaczmarz method for nonlinear systems of equations