Abstract:The recent years have witnessed advances in parallel algorithms for large-scale optimization problems. Notwithstanding the demonstrated success, existing algorithms that parallelize over features are usually limited by divergence issues under high parallelism or require data preprocessing to alleviate these problems. In this paper, we propose a Parallel Coordinate Descent algorithm using approximate Newton steps (PCDN) that is guaranteed to converge globally without data preprocessing. The key component of the PCDN algorithm is the high-dimensional line search, which guarantees the global convergence with high parallelism. The PCDN algorithm randomly partitions the feature set into b subsets/bundles of size P , and sequentially processes each bundle by first computing the descent directions for each feature in parallel and then conducting P -dimensional line search to compute the step size. We show that: 1) the PCDN algorithm is guaranteed to converge globally despite increasing parallelism and 2) the PCDN algorithm converges to the specified accuracy \epsilon within the limited iteration number of T_\epsilon , and T_\epsilon decreases with increasing parallelism. In addition, the data transfer and synchronization cost of the P -dimensional line search can be minimized by maintaining intermediate quantities. For concreteness, the proposed PCDN algorithm is applied to L_{1} -regularized logistic regression and L_{1} -regularized L_{2} -loss support vector machine problems. Experimental evaluations on seven benchmark data sets show that the PCDN algorithm exploits parallelism well and outperforms the state-of-the-art methods.

Dual Newton Proximal Point Algorithm for Solution Paths of the L1-Regularized Logistic Regression

A PROXIMAL POINT DUAL NEWTON ALGORITHM FOR SOLVING GROUP GRAPHICAL LASSO PROBLEMS

A Fast Dual Projected Newton Method for L1-Regularized Least Squares

A Proximal Point Algorithm For Log-Determinant Optimization With Group Lasso Regularization

A fast dual projected newton method for ℓ1-regularized least squares

Parallel Coordinate Descent Newton Method for Efficient L1 -Regularized Loss Minimization.

A Proximal Iteratively Regularized Gauss–Newton Method for Nonlinear Inverse Problems

Parallel Coordinate Descent Newton for Large-scale L1-Regularized Minimization.

A REGULARIZED NEWTON METHOD FOR <i>l<sub>q</sub></i>-ORM COMPOSITE OPTIMIZATION PROBLEMS

A Smoothing Proximal Gradient Algorithm for Nonsmooth Convex Regression with Cardinality Penalty.

Proximal Gradient Method with Extrapolation and Line Search for a Class of Nonconvex and Nonsmooth Problems

Parameterized proximal-gradient algorithms for L1/L2 sparse signal recovery

A Finite Newton Algorithm for Non-degenerate Piecewise Linear Systems.

PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networks

A Regularized Newton Method for \({\boldsymbol{\ell}}_{q}\) -Norm Composite Optimization Problems.

Proximal Quasi-Newton for Computationally Intensive L1-regularized M-estimators

Nonconvex Sparse Logistic Regression Via Proximal Gradient Descent.

A Proximal Gradient Method for Regularized Deep Neural Networks

A globally convergent regularized Newton method for $\ell_q$-norm composite optimization problems

A Proximal-Proximal Majorization-Minimization Algorithm for Nonconvex Rank Regression Problems.

A regularized Newton method for $\ell_q$-norm composite optimization problems