Abstract:In recent years, stochastic gradient descent (SGD) methods and randomized linear algebra (RLA) algorithms have been applied to many large-scale problems in machine learning and data analysis. SGD methods are easy to implement and applicable to a wide range of convex optimization problems. In contrast, RLA algorithms provide much stronger performance guarantees but are applicable to a narrower class of problems. We aim to bridge the gap between these two methods in solving constrained overdetermined linear regression problems-e.g., ℓ2 and ℓ1 regression problems. We propose a hybrid algorithm named pwSGD that uses RLA techniques for preconditioning and constructing an importance sampling distribution, and then performs an SGD-like iterative process with weighted sampling on the preconditioned system.By rewriting a deterministic ℓ p regression problem as a stochastic optimization problem, we connect pwSGD to several existing ℓ p solvers including RLA methods with algorithmic leveraging (RLA for short).We prove that pwSGD inherits faster convergence rates that only depend on the lower dimension of the linear system, while maintaining low computation complexity. Such SGD convergence rates are superior to other related SGD algorithm such as the weighted randomized Kaczmarz algorithm.Particularly, when solving ℓ1 regression with size n by d, pwSGD returns an approximate solution with ε relative error in the objective value in 𝒪(log n·nnz(A)+poly(d)/ε2) time. This complexity is uniformly better than that of RLA methods in terms of both ε and d when the problem is unconstrained. In the presence of constraints, pwSGD only has to solve a sequence of much simpler and smaller optimization problem over the same constraints. In general this is more efficient than solving the constrained subproblem required in RLA.For ℓ2 regression, pwSGD returns an approximate solution with ε relative error in the objective value and the solution vector measured in prediction norm in 𝒪(log n·nnz(A)+poly(d) log(1/ε)/ε) time. We show that for unconstrained ℓ2 regression, this complexity is comparable to that of RLA and is asymptotically better over several state-of-the-art solvers in the regime where the desired accuracy ε, high dimension n and low dimension d satisfy d ≥ 1/ε and n ≥ d2/ε. We also provide lower bounds on the coreset complexity for more general regression problems, indicating that still new ideas will be needed to extend similar RLA preconditioning ideas to weighted SGD algorithms for more general regression problems. Finally, the effectiveness of such algorithms is illustrated numerically on both synthetic and real datasets, and the results are consistent with our theoretical findings and demonstrate that pwSGD converges to a medium-precision solution, e.g., ε = 10-3, more quickly.

GSDAR: a Fast Newton Algorithm for ℓ _0 Regularized Generalized Linear Models with Statistical Guarantee

A Support Detection and Root Finding Approach for Learning High-dimensional Generalized Linear Models

Generalized Newton–Raphson Algorithm for High Dimensional LASSO Regression

A constructive approach to L

ℓ 0 -Regularized High-Dimensional Accelerated Failure Time Model.

A Constructive Approach to High-dimensional Regression

$\ell_1$-Regularized Generalized Least Squares

Fast Saddle-Point Algorithm for Generalized Dantzig Selector and FDR Control with the Ordered l1-Norm

Globally Convergent Newton Methods for Ill-conditioned Generalized Self-concordant Losses

Generalized Dantzig Selector: Application to the k-support norm

Semi-Smooth Newton Algorithm for Non-Convex Penalized Linear Regression

Generalized Metric Subregularity with Applications to High-Order Regularized Newton Methods

Regularization Effect of Fast Gradient Sign Method and its Generalization

SNAP: A semismooth Newton algorithm for pathwise optimization with optimal local convergence rate and oracle properties

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

Accelerated gradient methods for sparse statistical learning with nonconvex penalties

Adaptive Noisy Data Augmentation for Regularized Estimation and Inference in Generalized Linear Models

A generalization of regularized dual averaging and its dynamics

A Stochastic Iteratively Regularized Gauss-Newton Method

Weighted SGD for ℓ p Regression with Randomized Preconditioning

Generalized Newton Method with Positive Definite Regularization for Nonsmooth Optimization Problems with Nonisolated Solutions