Abstract:In large-scale unconstrained optimization algorithms such as limited memory BFGS (LBFGS), a common subproblem is a line search minimizing the loss function along a descent direction. Commonly used line searches iteratively find an approximate solution for which the Wolfe conditions are satisfied, typically requiring multiple function and gradient evaluations per line search, which is expensive in parallel due to communication requirements. In this paper we propose a new line search approach for cases where the loss function is analytic, as in least squares regression, logistic regression, or low rank matrix factorization. We approximate the loss function by a truncated Taylor polynomial, whose coefficients may be computed efficiently in parallel with less communication than evaluating the gradient, after which this polynomial may be minimized with high accuracy in a neighbourhood of the expansion point. Our Polynomial Expansion Line Search (PELS) was implemented in the Apache Spark framework and used to accelerate the training of a logistic regression model on binary classification datasets from the LIBSVM repository with LBFGS and the Nonlinear Conjugate Gradient (NCG) method. In large-scale numerical experiments in parallel on a 16-node cluster with 256 cores using the URL, KDDA, and KDDB datasets, the PELS approach produced significant convergence improvements compared to the use of classical Wolfe line searches. For example, to reach the final training label prediction accuracies, LBFGS using PELS had speedup factors of 1.8--2 over LBFGS using a Wolfe line search, measured by both the number of iterations and the time required, due to the better accuracy of step sizes computed in the line search. PELS has the potential to significantly accelerate large-scale regression and factorization computations, and is applicable to continuous optimization problems with smooth loss functions.

LSEMINK: A Modified Newton-Krylov Method for Log-Sum-Exp Minimization

Log-sum-exp Optimization based on Continuous Piecewise Linearization Techniques

A Multilevel Method for Self-Concordant Minimization

A Levenberg-Marquardt Method for Nonsmooth Regularized Least Squares

SNAP: A semismooth Newton algorithm for pathwise optimization with optimal local convergence rate and oracle properties

An efficient active-set method with applications to sparse approximations and risk minimization

A Semismooth-Newton's-Method-Based Linearization and Approximation Approach for Kernel Support Vector Machines

A polynomial expansion line search for large-scale unconstrained minimization of smooth L2-regularized loss functions, with implementation in Apache Spark

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

A Sparse Semismooth Newton Based Proximal Majorization-Minimization Algorithm for Nonconvex Square-Root-loss Regression Problems

Feature Screening Strategy for Non-Convex Sparse Logistic Regression with Log Sum Penalty

Adaptive Linearized Alternating Direction Method of Multipliers for Non-Convex Compositely Regularized Optimization Problems

A Learned Proximal Alternating Minimization Algorithm and Its Induced Network for a Class of Two-block Nonconvex and Nonsmooth Optimization

A trust region-type normal map-based semismooth Newton method for nonsmooth nonconvex composite optimization

Revisiting Sub-sampled Newton Methods

Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence

A nonlinear Lagrangian method based on Log-Sigmoid function for nonconvex semidefinite programming

Linearized Alternating Direction Method of Multipliers for Constrained Nonconvex Regularized Optimization.

Supervised Descent Method for Solving Nonlinear Least Squares Problems in Computer Vision

Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimization