Abstract:Excessive computational cost for learning large data and streaming data can be alleviated by using stochastic algorithms, such as stochastic gradient descent and its variants. Recent advances improve stochastic algorithms on convergence speed, adaptivity and structural awareness. However, distributional aspects of these new algorithms are poorly understood, especially for structured parameters. To develop statistical inference in this case, we propose a class of generalized regularized dual averaging (gRDA) algorithms with constant step size, which improves RDA (Xiao, 2010; Flammarion and Bach, 2017). Weak convergence of gRDA trajectories are studied, and as a consequence, for the first time in the literature, the asymptotic distributions for online l1 penalized problems become available. These general results apply to both convex and non-convex differentiable loss functions, and in particular, recover the existing regret bound for convex losses (Nemirovski et al., 2009). As important applications, statistical inferential theory on online sparse linear regression and online sparse principal component analysis are developed, and are supported by extensive numerical analysis. Interestingly, when gRDA is properly tuned, support recovery and central limiting distribution (with mean zero) hold simultaneously in the online setting, which is in contrast with the biased central limiting distribution of batch Lasso (Knight and Fu, 2000). Technical devices, including weak convergence of stochastic mirror descent, are developed as by-products with independent interest. Preliminary empirical analysis of modern image data shows that learning very sparse deep neural networks by gRDA does not necessarily sacrifice testing accuracy.

Lepskii Principle for Distributed Kernel Ridge Regression

Distributed Kernel Ridge Regression with Communications.

Adaptive Distributed Kernel Ridge Regression: A Feasible Distributed Learning Scheme for Data Silos

Distributed Semi-Supervised Learning with Kernel Ridge Regression

Distributed Learning with Indefinite Kernels

Distributed Learning With Dependent Samples

A Communication-Efficient Distributed Algorithm for Kernel Principal Component Analysis.

Debiased distributed learning for sparse partial linear models in high dimensions

Decentralized Kernel Ridge Regression Based on Data-Dependent Random Feature

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

Online Kernel Learning with a Near Optimal Sparsity Bound

A Duality Analysis of Kernel Ridge Regression in the Noiseless Regime

Learning Theory of Distributed Regression with Bias Corrected Regularization Kernel Network

COKE: Communication-Censored Kernel Learning for Decentralized Non-parametric Learning.

A generalization of regularized dual averaging and its dynamics

Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates

A Kernel Perspective on Distillation-based Collaborative Learning

Privacy-Preserving Distributed Linear Regression on High-Dimensional Data

On Distributed Exact Sparse Linear Regression over Networks

How Many Machines Can We Use in Parallel Computing for Kernel Ridge Regression?

Unified algorithms for distributed regularized linear regression model