Abstract:The recent years have witnessed advances in parallel algorithms for large-scale optimization problems. Notwithstanding the demonstrated success, existing algorithms that parallelize over features are usually limited by divergence issues under high parallelism or require data preprocessing to alleviate these problems. In this paper, we propose a Parallel Coordinate Descent algorithm using approximate Newton steps (PCDN) that is guaranteed to converge globally without data preprocessing. The key component of the PCDN algorithm is the high-dimensional line search, which guarantees the global convergence with high parallelism. The PCDN algorithm randomly partitions the feature set into b subsets/bundles of size P , and sequentially processes each bundle by first computing the descent directions for each feature in parallel and then conducting P -dimensional line search to compute the step size. We show that: 1) the PCDN algorithm is guaranteed to converge globally despite increasing parallelism and 2) the PCDN algorithm converges to the specified accuracy \epsilon within the limited iteration number of T_\epsilon , and T_\epsilon decreases with increasing parallelism. In addition, the data transfer and synchronization cost of the P -dimensional line search can be minimized by maintaining intermediate quantities. For concreteness, the proposed PCDN algorithm is applied to L_{1} -regularized logistic regression and L_{1} -regularized L_{2} -loss support vector machine problems. Experimental evaluations on seven benchmark data sets show that the PCDN algorithm exploits parallelism well and outperforms the state-of-the-art methods.

A Finite Newton Algorithm for Non-degenerate Piecewise Linear Systems.

Nondegenerate Piecewise Linear Systems: A Finite Newton Algorithm and Applications in Machine Learning

Parallel Coordinate Descent Newton Method for Efficient L1 -Regularized Loss Minimization.

The Existence the Solution of Nonlinear Discrete Schemes and Convergence of a Linearized Iterative Method for time-dependent PNP Equations

Newton iterations in implicit time-stepping scheme for differential linear complementarity systems

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

Parallel Coordinate Descent Newton for Large-scale L1-Regularized Minimization.

Newton Informed Neural Operator for Computing Multiple Solutions of Nonlinear Partials Differential Equations

Penalized Partial Least Squares Based on B-Splines Transformations

ON NEWTON-PSS METHODS FOR THE SYSTEM OF NONLINEAR EQUATIONS

Dual Newton Proximal Point Algorithm for Solution Paths of the L1-Regularized Logistic Regression

Nonnegative Iterative Reweighted Method for Sparse Linear Complementarity Problem

On the Finite Convergence of Newton-type Methods for P 0 Affine Variational Inequalities

Plug‐and‐play algorithms for convex non‐convex regularization: Convergence analysis and applications

A Solution Approach to the Nonconvex Piecewise Linear Network Flow Problems

Semi-Smooth Newton Algorithm for Non-Convex Penalized Linear Regression

A Fast Scale-Invariant Algorithm for Non-negative Least Squares with Non-negative Data

A Novel Nonlinear Functional Expansion Based PLS (FEPLS) and Its Soft Sensor Application

Coderivative-Based Newton Methods with Wolfe Linesearch for Nonsmooth Optimization

Revisiting Sub-sampled Newton Methods

A Stochastic Newton Method for Nonlinear Equations