Abstract:Sparsity regularized loss minimization problems play an important role in various fields including machine learning, data mining, and modern statistics. Proximal gradient descent method and coordinate descent method are the most popular approaches to solving the minimization problem. Although existing methods can achieve implicit model identification, aka support set identification, in a finite number of iterations, these methods still suffer from huge computational costs and memory burdens in high-dimensional scenarios. The reason is that the support set identification in these methods is implicit and thus cannot explicitly identify the low-complexity structure in practice, namely, they cannot discard useless coefficients of the associated features to achieve algorithmic acceleration via dimension reduction. To address this challenge, we propose a novel accelerated doubly stochastic gradient descent (ADSGD) method for sparsity regularized loss minimization problems, which can reduce the number of block iterations by eliminating inactive coefficients during the optimization process and eventually achieve faster explicit model identification and improve the algorithm efficiency. Theoretically, we first prove that ADSGD can achieve a linear convergence rate and lower overall computational complexity. More importantly, we prove that ADSGD can achieve a linear rate of explicit model identification. Numerically, experimental results on benchmark datasets confirm the efficiency of our proposed method.

Biased Compensation Adaptive Gradient Algorithm for Rational Model with Time-Delay Using Self-Organising Maps

Eliminating Ratio Bias for Gradient-based Simulated Parameter Estimation

Hierarchical Gradient Based Iterative Parameter Estimation Algorithm for Multivariable Output Error Moving Average Systems

Additional fractional gradient descent identification algorithm based on multi-innovation principle for autoregressive exogenous models

A Generalized Multivariate Grey Prediction Model DGM(1, N, I) with Unknown Time Delays

Iterative Parameter Identification for Time-delay Nonlinear Rational Models via L1-regularized Least Squares

Two‐stage auxiliary model gradient‐based iterative algorithm for the input nonlinear controlled autoregressive system with variable‐gain nonlinearity

Adaptive Multi-Innovation Gradient Identification Algorithms for a Controlled Autoregressive Autoregressive Moving Average Model

Reducing Bias in Deep Learning Optimization: The RSGDM Approach

Riemannian Stochastic Hybrid Gradient Algorithm for Nonconvex Optimization

Accelerated Nonconvex ADMM with Self-Adaptive Penalty for Rank-Constrained Model Identification

Adaptive Gradient Descent Algorithm for Networked Control Systems Using Redundant Rule

Stochastic Modified Equations and Adaptive Stochastic Gradient Algorithms

Auxiliary model maximum likelihood gradient‐based iterative identification for feedback nonlinear systems

Gradient-based Stochastic Extremum Seeking for Static Maps with Delays

An Accelerated Doubly Stochastic Gradient Method with Faster Explicit Model Identification

Two Time-Scale Gradient Approximation Algorithm For Adaptive Markov Reward Processes

Adaptive Accelerated Proximal Gradient Algorithm for Auto-Regressive Exogenous Models with Outliers

A Temporal-Difference Approach to Policy Gradient Estimation

Multi-innovation stochastic gradient identification method

The Nesterov accelerated gradient algorithm for Auto-Regressive Exogenous models with random lost measurements: interpolation method and auxiliary model method