Abstract:In large-scale unconstrained optimization algorithms such as limited memory BFGS (LBFGS), a common subproblem is a line search minimizing the loss function along a descent direction. Commonly used line searches iteratively find an approximate solution for which the Wolfe conditions are satisfied, typically requiring multiple function and gradient evaluations per line search, which is expensive in parallel due to communication requirements. In this paper we propose a new line search approach for cases where the loss function is analytic, as in least squares regression, logistic regression, or low rank matrix factorization. We approximate the loss function by a truncated Taylor polynomial, whose coefficients may be computed efficiently in parallel with less communication than evaluating the gradient, after which this polynomial may be minimized with high accuracy in a neighbourhood of the expansion point. Our Polynomial Expansion Line Search (PELS) was implemented in the Apache Spark framework and used to accelerate the training of a logistic regression model on binary classification datasets from the LIBSVM repository with LBFGS and the Nonlinear Conjugate Gradient (NCG) method. In large-scale numerical experiments in parallel on a 16-node cluster with 256 cores using the URL, KDDA, and KDDB datasets, the PELS approach produced significant convergence improvements compared to the use of classical Wolfe line searches. For example, to reach the final training label prediction accuracies, LBFGS using PELS had speedup factors of 1.8--2 over LBFGS using a Wolfe line search, measured by both the number of iterations and the time required, due to the better accuracy of step sizes computed in the line search. PELS has the potential to significantly accelerate large-scale regression and factorization computations, and is applicable to continuous optimization problems with smooth loss functions.

A Data-Driven Line Search Rule for Support Recovery in High-Dimensional Data Analysis.

Novel Linear Search for Support Vector Machine Parameter Selection

A Variance Minimization Criterion to Feature Selection Using Laplacian Regularization

A Improve Direct Path Seeking Algorithm for L1/2 Regularization, with Application to Biological Feature Selection

Computationally Efficient and Statistically Optimal Robust High-Dimensional Linear Regression

A Support Detection and Root Finding Approach for Learning High-dimensional Generalized Linear Models

Efficient Low-Rank Semidefinite Programming With Robust Loss Functions

Improving Sparsity and Scalability in Regularized Nonconvex Truncated-Loss Learning Problems

An Efficient Greedy Search Algorithm for High-Dimensional Linear Discriminant Analysis

Robust RLS Via the Nonconvex Sparsity Prompting Penalties of Outlier Components.

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

Recovery Discrimination for Nonperforming Loan: A Support Vector Machine Method with Variable Selection

Learning with Selected Features

On low-rank regularized least squares for scalable nonlinear classification

Generalized Newton–Raphson Algorithm for High Dimensional LASSO Regression

Sparse Support Recovery with Non-smooth Loss Functions

A Constructive Approach to High-dimensional Regression

A Tuning-free Robust and Efficient Approach to High-dimensional Regression

A polynomial expansion line search for large-scale unconstrained minimization of smooth L2-regularized loss functions, with implementation in Apache Spark

Recovery Discrimination Based on Optimized-Variables Support Vector Machine for Nonperforming Loan

A Robust Recovery Algorithm with Smoothing Strategies.