Abstract:In this paper we consider large-scale smooth optimization problems with multiple linear coupled constraints. Due to the non-separability of the constraints, arbitrary random sketching would not be guaranteed to work. Thus, we first investigate necessary and sufficient conditions for the sketch sampling to have well-defined algorithms. Based on these sampling conditions we developed new sketch descent methods for solving general smooth linearly constrained problems, in particular, random sketch descent and accelerated random sketch descent methods. From our knowledge, this is the first convergence analysis of random sketch descent algorithms for optimization problems with multiple non-separable linear constraints. For the general case, when the objective function is smooth and non-convex, we prove for the non-accelerated variant sublinear rate in expectation for an appropriate optimality measure. In the smooth convex case, we derive for both algorithms, non-accelerated and accelerated random sketch descent, sublinear convergence rates in the expected values of the objective function. Additionally, if the objective function satisfies a strong convexity type condition, both algorithms converge linearly in expectation. In special cases, where complexity bounds are known for some particular sketching algorithms, such as coordinate descent methods for optimization problems with a single linear coupled constraint, our theory recovers the best-known bounds. We also show that when random sketch is sketching the coordinate directions randomly produces better results than the fixed selection rule. Finally, we present some numerical examples to illustrate the performances of our new algorithms.

On convergence of a sketch-and-project method for the matrix equation

A sketch-and-project method for solving the matrix equation AXB = C

Convergence Analysis of Projected Gradient Descent for Schatten-p Nonconvex Matrix Recovery

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

The convergence of conjugate gradient method with nonmonotone line search

Convergence of projected Landweber iteration for matrix rank minimization

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Convergence results on iteration algorithms to linear systems.

A Revised Conjugate Gradient Projection Algorithm for Inequality Constrained Optimizations

Convergence Behavior of Gauss-Newton's Method and Extensions of the Smale Point Estimate Theory.

Convergence Characterization of an Iterative Algorithm for Continuous Coupled Lyapunov Matrix Equations

Sketch-and-Project Meets Newton Method: Global $\mathcal O(k^{-2})$ Convergence with Low-Rank Updates

Convergence of generalized alternating iterative methods

On Two Iteration Methods for the Quadratic Matrix Equations

A hybrid three-term conjugate gradient projection method for constrained nonlinear monotone equations with applications

On block Gaussian sketching for the Kaczmarz method

Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence

Efficient Projected Gradient Methods for L 0 Constrained Optimization

Randomized sketch descent methods for non-separable linearly constrained optimization

Randomized Iterative Methods for Linear Systems

Gradient-based Maximal Convergence Rate Iterative Method for Solving Linear Matrix Equations