Abstract:We develop a simple routine unifying the analysis of several important recently-developed stochastic optimization methods including SAGA, Finito, and stochastic dual coordinate ascent (SDCA). First, we show an intrinsic connection between stochastic optimization methods and dynamic jump systems, and propose a general jump system model for stochastic optimization methods. Our proposed model recovers SAGA, SDCA, Finito, and SAG as special cases. Then we combine jump system theory with several simple quadratic inequalities to derive sufficient conditions for convergence rate certifications of the proposed jump system model under various assumptions (with or without individual convexity, etc). The derived conditions are linear matrix inequalities (LMIs) whose sizes roughly scale with the size of the training set. We make use of the symmetry in the stochastic optimization methods and reduce these LMIs to some equivalent small LMIs whose sizes are at most 3 by 3. We solve these small LMIs to provide analytical proofs of new convergence rates for SAGA, Finito and SDCA (with or without individual convexity). We also explain why our proposed LMI fails in analyzing SAG. We reveal a key difference between SAG and other methods, and briefly discuss how to extend our LMI analysis for SAG. An advantage of our approach is that the proposed analysis can be automated for a large class of stochastic methods under various assumptions (with or without individual convexity, etc).

Improved Least Square Method Based on Theory of Programming and Application in Parameters Estimating of Markov Jumping Systems

Mean-square stability of stochastic Markovian jumping systems with variable delay

A New Algorithm For Unconstrained Optimization Problem With The Form Of Sum Of Squares Minimization

Modular‐based Adaptive Control of Uncertain Nonlinear Time‐varying Systems with Piecewise Jumping Parameters

Reshaping of the Probability Density Function of Nonlinear Stochastic Systems Against Abrupt Changes

Optimal Vibration Control of a Class of Nonlinear Stochastic Systems with Markovian Jump

A Unified Analysis of Stochastic Optimization Methods Using Jump System Theory and Quadratic Constraints

Observer design for semi-Markov jump systems with incremental quadratic constraints

A Model-Free Iteration Algorithm for Markov Jump Linear Systems Based on Gauss-Seidel Method

Improved Results on H∞ Model Reduction for Markovian Jump Systems with Partly Known Transition Probabilities

Stabilization and H∞ Control of Two-Dimensional Markovian Jump Systems

Probability rate optimization of positive Markov jump linear systems via DC programming

Optimization For Controlled Jump Rates Of Jlqg Problem

Jump Linear Quadratic Gaussian Problem for A Class of Nonhomogeneous Markov Jump Linear Systems

A Robust Suppress Of Stochastic Nonlinear Systems With Markovian Jump Parameters

Comparison theorems for multi-dimensional BSDEs with jumps and applications to constrained stochastic linear-quadratic control

Reachable Set Estimation for Markov Jump LPV Systems with Time Delays.

Robust Kalman Filter of Discrete-Time Markovian Jump System with Parameter and Noise Uncertainty

The Stability and Stabilization of Markov Jump Linear Systems with Regime Cluster

Reachable set estimation of delayed Markovian jump neural networks via variables-augmented-based free-weighting-matrices method

A Successive over Relaxation Implicit Iterative Algorithm for Solving Stochastic Linear Systems with Markov Jumps