Abstract:In this work, we study a novel class of projection-based algorithms for linearly constrained problems (LCPs) which have a lot of applications in statistics, optimization, and machine learning. Conventional primal gradient-based methods for LCPs call a projection after each (stochastic) gradient descent, resulting in that the required number of projections equals that of gradient descents (or total iterations). Motivated by the recent progress in distributed optimization, we propose the delayed projection technique that calls a projection once for a while, lowering the projection frequency and improving the projection efficiency. Accordingly, we devise a series of stochastic methods for LCPs using the technique, including a variance reduced method and an accelerated one. We theoretically show that it is feasible to improve projection efficiency in both strongly convex and generally convex cases. Our analysis is simple and unified and can be easily extended to other methods using delayed projections. When applying our new algorithms to federated optimization, a newfangled and privacy-preserving subfield in distributed optimization, we obtain not only a variance reduced federated algorithm with convergence rates better than previous works, but also the first accelerated method able to handle data heterogeneity inherent in federated optimization.

A Delayed Neural Network Method for Solving Convex Optimization Problems.

A Delayed Neural Network for Solving a Class of Constrained Pseudoconvex Optimizations

Neural Network Solution for General Nonlinear Optimization Problems

A New Delayed Projection Neural Network for Solving Linear Variational Inequalities and Quadratic Optimization Problems

A Neutral-Type Delayed Projection Neural Network for Solving Quadratic Programming Problems

A Novel Neural Network for Nonlinear Convex Programming

Improved Results on Solving Quadratic Programming Problems with Delayed Neural Network

A Neural Network Model for General Minimax Problem

A Neural Network for Solving Nonlinear Convex Programming with Linear Equality and Bounded Constraints

A discrete-time neural network for optimization problems with hybrid constraints.

Neural Network for Solving Constrained Convex Optimization Problems with Global Attractivity

A Neurodynamic Approach to Convex Optimization Problems with General Constraint.

A Project Neural Network for Solving Degenerate Convex Quadratic Program

Neural Network for Nonsmooth Pseudoconvex Optimization with General Convex Constraints

A fast algorithm for solving large scale nonlinear optimization problems using RNN

Delay-independent and delay-dependent stability of a novel delayed neural network

Convergence Analysis of Discrete-Time Simplified Dual Neural Network for Solving Convex Quadratic Programming Problems

Solving for a Quadratic Programming with a Quadratic Constraint Based on a Neural Network Frame

A Recurrent Neural Network for Solving Nonconvex Optimization Problems

Delayed Projection Techniques for Linearly Constrained Problems: Convergence Rates, Acceleration, and Applications

A novel neural dynamical approach to convex quadratic program and its efficient applications.