Abstract:We establish an equivalence between the $\ell_2$-regularized solution path for a convex loss function, and the solution of an ordinary differentiable equation (ODE). Importantly, this equivalence reveals that the solution path can be viewed as the flow of a hybrid of gradient descent and Newton method applying to the empirical loss, which is similar to a widely used optimization technique called trust region method. This provides an interesting algorithmic view of $\ell_2$ regularization, and is in contrast to the conventional view that the $\ell_2$ regularization solution path is similar to the gradient flow of the empirical <a class="link-external link-http" href="http://loss.New" rel="external noopener nofollow">this http URL</a> path-following algorithms based on homotopy methods and numerical ODE solvers are proposed to numerically approximate the solution path. In particular, we consider respectively Newton method and gradient descent method as the basis algorithm for the homotopy method, and establish their approximation error rates over the solution path. Importantly, our theory suggests novel schemes to choose grid points that guarantee an arbitrarily small suboptimality for the solution path. In terms of computational cost, we prove that in order to achieve an $\epsilon$-suboptimality for the entire solution path, the number of Newton steps required for the Newton method is $\mathcal O(\epsilon^{-1/2})$, while the number of gradient steps required for the gradient descent method is $\mathcal O\left(\epsilon^{-1} \ln(\epsilon^{-1})\right)$. Finally, we use $\ell_2$-regularized logistic regression as an illustrating example to demonstrate the effectiveness of the proposed path-following algorithms.

Regularization properties of dual subgradient flow

Flow by Gauss Curvature to the $ L_p $ Dual Minkowski Problem

Regularized Stein Variational Gradient Flow

Subgradient-based Lavrentiev regularisation of monotone ill-posed problems

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

A primal-dual flow for affine constrained convex optimization

The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Program

Smoothing $\mathcal{L}^2$ gradients in iterative regularization

A duality based approach to the minimizing total variation flow in the space $H^{-s}$

Weighted Energy-Dissipation approach to semilinear gradient flows with state-dependent dissipation

Structure Preserving Primal Dual Methods for Gradient Flows with Nonlinear Mobility Transport Distances

Existence, uniqueness and $L^2_t (H_x ^2) \cap L^\infty_t (H^1_x) \cap H^1_t (L^2_x) $ regularity of the gradient flow of the Ambrosio-Tortorelli functional

Interplay between preconditioning and regularization for linear ill-posed problems solved by conjugate gradient. Application to optical flow estimation

A duality-based approach to gradient flows of linear growth functionals

Existence, uniqueness and L2t(H2x) ∩ L∞t(H1x) ∩ H1t(L2x) regularity of the gradient flow of the Ambrosio–Tortorelli functional

A new flow dynamic approach for Wasserstein gradient flows

Some Primal-Dual Theory for Subgradient Methods for Strongly Convex Optimization

An algorithmic view of $\ell_2$ regularization and some path-following algorithms

On Stochastic Primal-Dual Hybrid Gradient Approach for Compositely Regularized Minimization.

How to induce regularization in linear models: A guide to reparametrizing gradient flow

Gradient Flows for Regularized Stochastic Control Problems