Abstract:Sparsity regularized loss minimization problems play an important role in various fields including machine learning, data mining, and modern statistics. Proximal gradient descent method and coordinate descent method are the most popular approaches to solving the minimization problem. Although existing methods can achieve implicit model identification, aka support set identification, in a finite number of iterations, these methods still suffer from huge computational costs and memory burdens in high-dimensional scenarios. The reason is that the support set identification in these methods is implicit and thus cannot explicitly identify the low-complexity structure in practice, namely, they cannot discard useless coefficients of the associated features to achieve algorithmic acceleration via dimension reduction. To address this challenge, we propose a novel accelerated doubly stochastic gradient descent (ADSGD) method for sparsity regularized loss minimization problems, which can reduce the number of block iterations by eliminating inactive coefficients during the optimization process and eventually achieve faster explicit model identification and improve the algorithm efficiency. Theoretically, we first prove that ADSGD can achieve a linear convergence rate and lower overall computational complexity. More importantly, we prove that ADSGD can achieve a linear rate of explicit model identification. Numerically, experimental results on benchmark datasets confirm the efficiency of our proposed method.

Hybrid gradient descent algorithms for the exponential autoregressive model

Hybrid optimization algorithm for load model parameter identification

Improve recursive identification using multi-iteration

A Method of ARX Model Estimation of Errors-in-Variables Systems

A NEW IDENTIFICATION APPROACH FOR ARX MODELS

The Nesterov accelerated gradient algorithm for Auto-Regressive Exogenous models with random lost measurements: interpolation method and auxiliary model method

Adaptive Accelerated Proximal Gradient Algorithm for Auto-Regressive Exogenous Models with Outliers

Gradient-based Parameter Estimation for a Nonlinear Exponential Autoregressive Time-series Model by Using the Multi-innovation

Additional fractional gradient descent identification algorithm based on multi-innovation principle for autoregressive exogenous models

Convergence of Recursive Identification for ARMAX Process with Increasing Variances

Cauchy kernel correntropy‐based robust multi‐innovation identification method for the nonlinear exponential autoregressive model in non‐Gaussian environment

Fitting ARMA Time Series Models without Identification: A Proximal Approach

A novel expectation–maximization-based separable algorithm for parameter identification of RBF-AR model

Adaptive Multi-Innovation Gradient Identification Algorithms for a Controlled Autoregressive Autoregressive Moving Average Model

Two‐stage auxiliary model gradient‐based iterative algorithm for the input nonlinear controlled autoregressive system with variable‐gain nonlinearity

Maximum likelihood based multi‐innovation stochastic gradient identification algorithms for bilinear stochastic systems with ARMA noise

Hierarchical Quasi-Fractional Gradient Descent Method for Parameter Estimation of Nonlinear ARX Systems Using Key Term Separation Principle

Sequential Prediction-Error Method for Structural Identification

Two-stage Gradient-based Iterative Estimation Methods for Controlled Autoregressive Systems Using the Measurement Data

Hierarchical Gradient Based Iterative Parameter Estimation Algorithm for Multivariable Output Error Moving Average Systems

An Accelerated Doubly Stochastic Gradient Method with Faster Explicit Model Identification