Abstract:Averaging iterations of Stochastic Gradient Descent (SGD) have achieved empirical success in training deep learning models, such as Stochastic Weight Averaging (SWA), Exponential Moving Average (EMA), and LAtest Weight Averaging (LAWA). Especially, with a finite weight averaging method, LAWA can attain faster convergence and better generalization. However, its theoretical explanation is still less explored since there are fundamental differences between finite and infinite settings. In this work, we first generalize SGD and LAWA as Finite Weight Averaging (FWA) and explain their advantages compared to SGD from the perspective of optimization and generalization. A key challenge is the inapplicability of traditional methods in the sense of expectation or optimal values for infinite-dimensional settings in analyzing FWA's convergence. Second, the cumulative gradients introduced by FWA introduce additional confusion to the generalization analysis, especially making it more difficult to discuss them under different assumptions. Extending the final iteration convergence analysis to the FWA, this paper, under a convexity assumption, establishes a convergence bound $\mathcal{O}(\log\left(\frac{T}{k}\right)/\sqrt{T})$, where $k\in[1, T/2]$ is a constant representing the last $k$ iterations. Compared to SGD with $\mathcal{O}(\log(T)/\sqrt{T})$, we prove theoretically that FWA has a faster convergence rate and explain the effect of the number of average points. In the generalization analysis, we find a recursive representation for bounding the cumulative gradient using mathematical induction. We provide bounds for constant and decay learning rates and the convex and non-convex cases to show the good generalization performance of FWA. Finally, experimental results on several benchmarks verify our theoretical results.

Convergence Analysis of Weighted Stochastic Gradient Identification Algorithms Based on Latest‐Estimation for ARX Models

Weighted Hierarchical Stochastic Gradient Identification Algorithms for ARX Models

Latest Estimation Based Recursive Stochastic Gradient Identification Algorithms for Arx Models

Fractional-Based Stochastic Gradient Algorithms for Time-Delayed ARX Models.

Aitken-Based Stochastic Gradient Algorithm for ARX Models with Time Delay

Robust Standard Gradient Descent Algorithm for ARX Models Using Aitken Acceleration Technique.

Modified Kalman Filtering Based Multi-Step-length Gradient Iterative Algorithm for ARX Models with Random Missing Outputs

Gradient-Based Particle Filter Algorithm for an ARX Model with Nonlinear Communication Output

Maximum Likelihood Parameter Estimation for ARMAX Models Based on Stochastic Gradient Algorithm

Latest-Estimation Based Hierarchical Recursive Extended Least Squares Algorithm for Armax Model

A NEW IDENTIFICATION APPROACH FOR ARX MODELS

Generalization Analysis of Stochastic Weight Averaging with General Sampling

Parameter Identification of ARX Models Based on Modified Momentum Gradient Descent Algorithm

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Global convergence of the EM algorithm for ARX models with uncertain communication channels

Modified Recursive Extended Least Squares Identification Algorithms

Recursive And Iterative Stochastic Gradient Algorithms Based On Two-Step Update Estimation For Wiener Systems

On the asymptotic rate of convergence of Stochastic Newton algorithms and their Weighted Averaged versions

A Unified Analysis for Finite Weight Averaging

Convergence of Recursive Identification for ARMAX Process with Increasing Variances

The Nesterov accelerated gradient algorithm for Auto-Regressive Exogenous models with random lost measurements: interpolation method and auxiliary model method