Abstract:Averaging iterations of Stochastic Gradient Descent (SGD) have achieved empirical success in training deep learning models, such as Stochastic Weight Averaging (SWA), Exponential Moving Average (EMA), and LAtest Weight Averaging (LAWA). Especially, with a finite weight averaging method, LAWA can attain faster convergence and better generalization. However, its theoretical explanation is still less explored since there are fundamental differences between finite and infinite settings. In this work, we first generalize SGD and LAWA as Finite Weight Averaging (FWA) and explain their advantages compared to SGD from the perspective of optimization and generalization. A key challenge is the inapplicability of traditional methods in the sense of expectation or optimal values for infinite-dimensional settings in analyzing FWA's convergence. Second, the cumulative gradients introduced by FWA introduce additional confusion to the generalization analysis, especially making it more difficult to discuss them under different assumptions. Extending the final iteration convergence analysis to the FWA, this paper, under a convexity assumption, establishes a convergence bound $\mathcal{O}(\log\left(\frac{T}{k}\right)/\sqrt{T})$, where $k\in[1, T/2]$ is a constant representing the last $k$ iterations. Compared to SGD with $\mathcal{O}(\log(T)/\sqrt{T})$, we prove theoretically that FWA has a faster convergence rate and explain the effect of the number of average points. In the generalization analysis, we find a recursive representation for bounding the cumulative gradient using mathematical induction. We provide bounds for constant and decay learning rates and the convex and non-convex cases to show the good generalization performance of FWA. Finally, experimental results on several benchmarks verify our theoretical results.

Convergence Analysis of Weighted Stochastic Gradient Identification Algorithms Based on Latest‐Estimation for ARX Models

Improve recursive identification using multi-iteration

Robust Iterative Algorithm of Weighted Least Squares Support Vector Machine and Its Application in Spectral Analysis

A Method of ARX Model Estimation of Errors-in-Variables Systems

A NEW IDENTIFICATION APPROACH FOR ARX MODELS

Generalization Analysis of Stochastic Weight Averaging with General Sampling

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Convergence analysis of the particle swarm optimization with stochastic inertia weight

On the asymptotic rate of convergence of Stochastic Newton algorithms and their Weighted Averaged versions

A Unified Analysis for Finite Weight Averaging

Convergence of Recursive Identification for ARMAX Process with Increasing Variances

The Nesterov accelerated gradient algorithm for Auto-Regressive Exogenous models with random lost measurements: interpolation method and auxiliary model method

Optimized convergence of stochastic gradient descent by weighted averaging

Convergence of SGD with momentum in the nonconvex case: A time window-based analysis

Gaussian Weighting Reversion Strategy For Accurate On-Line Portfolio Selection

A Sharp Convergence Rate for the Asynchronous Stochastic Gradient Descent

Gaussian Weighting Reversion Strategy for Accurate Online Portfolio Selection

Maximum likelihood based multi‐innovation stochastic gradient identification algorithms for bilinear stochastic systems with ARMA noise

Optimal Adaptive Controller for Stochastic Systems Based on Weighted Least-squares Algorithm

Servo turntable adaptive step size momentum projection identification algorithm based on ARX model

New Insight into Hybrid Stochastic Gradient Descent: Beyond With-Replacement Sampling and Convexity.