Abstract:Sparsity regularized loss minimization problems play an important role in various fields including machine learning, data mining, and modern statistics. Proximal gradient descent method and coordinate descent method are the most popular approaches to solving the minimization problem. Although existing methods can achieve implicit model identification, aka support set identification, in a finite number of iterations, these methods still suffer from huge computational costs and memory burdens in high-dimensional scenarios. The reason is that the support set identification in these methods is implicit and thus cannot explicitly identify the low-complexity structure in practice, namely, they cannot discard useless coefficients of the associated features to achieve algorithmic acceleration via dimension reduction. To address this challenge, we propose a novel accelerated doubly stochastic gradient descent (ADSGD) method for sparsity regularized loss minimization problems, which can reduce the number of block iterations by eliminating inactive coefficients during the optimization process and eventually achieve faster explicit model identification and improve the algorithm efficiency. Theoretically, we first prove that ADSGD can achieve a linear convergence rate and lower overall computational complexity. More importantly, we prove that ADSGD can achieve a linear rate of explicit model identification. Numerically, experimental results on benchmark datasets confirm the efficiency of our proposed method.

Accelerating Variable Projection Algorithm by Aitken Technique.

Recursive Variable Projection Algorithm for a Class of Separable Nonlinear Models

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Variable projection algorithms with sparse constraint for separable nonlinear models

Variable Projection Algorithms: Theoretical Insights and A Novel Approach for Problems with Large Residual

Multi-innovation-based online variable projection algorithm for a class of nonlinear models

Online Identification of Nonlinear Systems With Separable Structure

Convergence Analysis of a Variable Projection Method for Regularized Separable Nonlinear Inverse Problems

Accelerated Sparse Linear Regression Via Random Projection.

A Variable Projection-Based Algorithm for Fault Detection and Diagnosis

Insights Into Algorithms for Separable Nonlinear Least Squares Problems

Variable projection methods for separable nonlinear inverse problems with general-form Tikhonov regularization

Anderson acceleration of derivative-free projection methods for constrained monotone nonlinear equations

An Accelerated Doubly Stochastic Gradient Method with Faster Explicit Model Identification

NLTGCR: A class of Nonlinear Acceleration Procedures based on Conjugate Residuals

Nonmonotone variable projection algorithms for matrix decomposition with missing data

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

Accelerated gradient methods for sparse statistical learning with nonconvex penalties

On a Dynamic Variant of the Iteratively Regularized Gauss-Newton Method with Sequential Data.

Asynchronous Accelerated Stochastic Gradient Descent.

Anderson Acceleration with Truncated Gram-Schmidt