Abstract:Excessive computational cost for learning large data and streaming data can be alleviated by using stochastic algorithms, such as stochastic gradient descent and its variants. Recent advances improve stochastic algorithms on convergence speed, adaptivity and structural awareness. However, distributional aspects of these new algorithms are poorly understood, especially for structured parameters. To develop statistical inference in this case, we propose a class of generalized regularized dual averaging (gRDA) algorithms with constant step size, which improves RDA (Xiao, 2010; Flammarion and Bach, 2017). Weak convergence of gRDA trajectories are studied, and as a consequence, for the first time in the literature, the asymptotic distributions for online l1 penalized problems become available. These general results apply to both convex and non-convex differentiable loss functions, and in particular, recover the existing regret bound for convex losses (Nemirovski et al., 2009). As important applications, statistical inferential theory on online sparse linear regression and online sparse principal component analysis are developed, and are supported by extensive numerical analysis. Interestingly, when gRDA is properly tuned, support recovery and central limiting distribution (with mean zero) hold simultaneously in the online setting, which is in contrast with the biased central limiting distribution of batch Lasso (Knight and Fu, 2000). Technical devices, including weak convergence of stochastic mirror descent, are developed as by-products with independent interest. Preliminary empirical analysis of modern image data shows that learning very sparse deep neural networks by gRDA does not necessarily sacrifice testing accuracy.

Deterministic convergence analysis for regularized long short-term memory and its application to regression and multi-classification problems

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Convergence of Gradient Method for A Fully Recurrent Neural Network

Convergence Analysis of Stochastic Gradient Descent with MCMC Estimators

Convergence Analysis of Regularized Elman Neural Networks under Relaxed Conditions

Convergence Analysis of an Adaptively Regularized Natural Gradient Method

Deterministic Gradient-Descent Learning of Linear Regressions: Adaptive Algorithms, Convergence Analysis and Noise Compensation

LLR: Learning Learning Rates by LSTM for Training Neural Networks.

A generalization of regularized dual averaging and its dynamics

Stability and convergence analysis of AdaGrad for non-convex optimization via novel stopping time-based techniques

A Unified Analysis of Stochastic Momentum Methods for Deep Learning

Convergence of Stochastic Gradient Descent in Deep Neural Network

Convergence Conditions of Online Regularized Statistical Learning in Reproducing Kernel Hilbert Space With Non-Stationary Data

A Convergent ADMM Framework for Efficient Neural Network Training

Convergence Analysis for Over-Parameterized Deep Learning

A Unified Analysis of Stochastic Momentum Methods for Deep Learning.

Long short-term memory with activation on gradient

Convergence of Batch Gradient Learning with Smoothing Regularization and Adaptive Momentum for Neural Networks

A Gram-Gauss-Newton Method Learning Overparameterized Deep Neural Networks for Regression Problems.

Estimation of Convergence Rate for Multi-Regression Learning Algorithm

Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization