Abstract:We establish an equivalence between the $\ell_2$-regularized solution path for a convex loss function, and the solution of an ordinary differentiable equation (ODE). Importantly, this equivalence reveals that the solution path can be viewed as the flow of a hybrid of gradient descent and Newton method applying to the empirical loss, which is similar to a widely used optimization technique called trust region method. This provides an interesting algorithmic view of $\ell_2$ regularization, and is in contrast to the conventional view that the $\ell_2$ regularization solution path is similar to the gradient flow of the empirical <a class="link-external link-http" href="http://loss.New" rel="external noopener nofollow">this http URL</a> path-following algorithms based on homotopy methods and numerical ODE solvers are proposed to numerically approximate the solution path. In particular, we consider respectively Newton method and gradient descent method as the basis algorithm for the homotopy method, and establish their approximation error rates over the solution path. Importantly, our theory suggests novel schemes to choose grid points that guarantee an arbitrarily small suboptimality for the solution path. In terms of computational cost, we prove that in order to achieve an $\epsilon$-suboptimality for the entire solution path, the number of Newton steps required for the Newton method is $\mathcal O(\epsilon^{-1/2})$, while the number of gradient steps required for the gradient descent method is $\mathcal O\left(\epsilon^{-1} \ln(\epsilon^{-1})\right)$. Finally, we use $\ell_2$-regularized logistic regression as an illustrating example to demonstrate the effectiveness of the proposed path-following algorithms.

A NEW NUMERICAL DIFFERENTIATION ALGORITHM WITH REGULARIZATION

Some Results on Numerical Divided Difference Formulas

Numerical Differentiation from A Viewpoint of Regularization Theory

First and Second Order Numerical Differentiation with Tikhonov Regularization

Numerical Differentiation for the Second Order Derivatives of Functions of Two Variables

A Numerical Differentiation Method and Its Application to Reconstruction of Discontinuity

Numerical Differentiation by a Tikhonov Regularization Method Based on the Discrete Cosine Transform

Detection of Irregular Points by Regularization in Numerical Differentiation and Application to Edge Detection

Some Substantial Modifications and Improvements for Derivative-Free Iterative Methods and Derivative-Free Transformation for Multiple Zeros

Presentation, error analysis and numerical experiments on a group of 1-step-ahead numerical differentiation formulas.

Numerical Differentiation and Its Applications

A Tikhonov Regularization Based Algorithm for Scattered Data with Random Noise

A Regularized Newton Method for Monotone Nonlinear Equations and Its Application

A data-driven adaptive regularization method and its applications

Regularization with two differential operators and its application to inverse problems

(Almost) Smooth Sailing: Towards Numerical Stability of Neural Networks Through Differentiable Regularization of the Condition Number

On regularized Radon-Nikodym differentiation

Generalized Newton Method with Positive Definite Regularization for Nonsmooth Optimization Problems with Nonisolated Solutions

An algorithmic view of $\ell_2$ regularization and some path-following algorithms

A Big Data Processing Technique Based on Tikhonov Regularization

A Dynamical Regularization Algorithm for Solving Inverse Source Problems of Elliptic Partial Differential Equations