Abstract:In recent years, stochastic gradient descent (SGD) methods and randomized linear algebra (RLA) algorithms have been applied to many large-scale problems in machine learning and data analysis. SGD methods are easy to implement and applicable to a wide range of convex optimization problems. In contrast, RLA algorithms provide much stronger performance guarantees but are applicable to a narrower class of problems. We aim to bridge the gap between these two methods in solving constrained overdetermined linear regression problems-e.g., ℓ2 and ℓ1 regression problems. We propose a hybrid algorithm named pwSGD that uses RLA techniques for preconditioning and constructing an importance sampling distribution, and then performs an SGD-like iterative process with weighted sampling on the preconditioned system.By rewriting a deterministic ℓ p regression problem as a stochastic optimization problem, we connect pwSGD to several existing ℓ p solvers including RLA methods with algorithmic leveraging (RLA for short).We prove that pwSGD inherits faster convergence rates that only depend on the lower dimension of the linear system, while maintaining low computation complexity. Such SGD convergence rates are superior to other related SGD algorithm such as the weighted randomized Kaczmarz algorithm.Particularly, when solving ℓ1 regression with size n by d, pwSGD returns an approximate solution with ε relative error in the objective value in 𝒪(log n·nnz(A)+poly(d)/ε2) time. This complexity is uniformly better than that of RLA methods in terms of both ε and d when the problem is unconstrained. In the presence of constraints, pwSGD only has to solve a sequence of much simpler and smaller optimization problem over the same constraints. In general this is more efficient than solving the constrained subproblem required in RLA.For ℓ2 regression, pwSGD returns an approximate solution with ε relative error in the objective value and the solution vector measured in prediction norm in 𝒪(log n·nnz(A)+poly(d) log(1/ε)/ε) time. We show that for unconstrained ℓ2 regression, this complexity is comparable to that of RLA and is asymptotically better over several state-of-the-art solvers in the regime where the desired accuracy ε, high dimension n and low dimension d satisfy d ≥ 1/ε and n ≥ d2/ε. We also provide lower bounds on the coreset complexity for more general regression problems, indicating that still new ideas will be needed to extend similar RLA preconditioning ideas to weighted SGD algorithms for more general regression problems. Finally, the effectiveness of such algorithms is illustrated numerically on both synthetic and real datasets, and the results are consistent with our theoretical findings and demonstrate that pwSGD converges to a medium-precision solution, e.g., ε = 10-3, more quickly.

Variable step-size convex regularized PRLS algorithms

Boosting the Partial Least Square Algorithm for Regression Modelling

A New Penalized Recursive Least Squares Method With a Variable Regularization Factor for Adaptive Sparse Filtering.

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Iterative-Promoting Variable Step-size Least Mean Square Algorithm For Adaptive Sparse Channel Estimation

Sparse Linear Regression: Sequential Convex Relaxation, Robust Restricted Null Space Property, and Variable Selection

Recursive Least Squares for Censored Regression

Sparse Estimation Via ℓ_q Optimization Method in High-Dimensional Linear Regression

Variable Partial-Update NLMS Algorithms with Data-Selective Updating.

Robust Variable Step Size Widely Linear Complex-Valued Least Mean M-estimate Adaptive Algorithm: Derivation and Performance Analysis

A Fast Accurate Two-stage Training Algorithm for L1-regularized CRFs with Heuristic Line Search Strategy.

Regularized Total Least Squares: Computational Aspects and Error Bounds.

Regularized Least-Mean-Square Algorithms

Weighted SGD for ℓ p Regression with Randomized Preconditioning

Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization

EM Based p-norm-like Constraint RLS Algorithm for Sparse System Identification

A Constant Step Stochastic Douglas-Rachford Algorithm with Application to Non Separable Regularizations

Regularized Barzilai-Borwein method

A New Variable Step Size LMS Algorithm with Application to Active Noise Control

Improving Sparsity and Scalability in Regularized Nonconvex Truncated-Loss Learning Problems

Partial least trimmed squares regression