Abstract:We introduce a variant of the proximal gradient method in which the quadratic term is diagonal but may be indefinite, and is safeguarded by a trust region. Our method is a special case of the proximal quasi-Newton trust-region method of <a class="link-https" data-arxiv-id="2103.15993v3" href="https://arxiv.org/abs/2103.15993v3">arXiv:2103.15993v3</a>. We provide closed-form solution of the step computation in certain cases where the nonsmooth term is separable and the trust region is defined in infinity norm, so that no iterative subproblem solver is required. Our analysis expands upon that of <a class="link-https" data-arxiv-id="2103.15993v3" href="https://arxiv.org/abs/2103.15993v3">arXiv:2103.15993v3</a> by generalizing the trust-region approach to problems with bound constraints. We provide an efficient open-source implementation of our method, named TRDH, in the Julia language in which Hessians approximations are given by diagonal quasi-Newton updates. TRDH evaluates one standard proximal operator and one indefinite proximal operator per iteration. We also analyze and implement a variant named iTRDH that performs a single indefinite proximal operator evaluation per iteration. We establish that iTRDH enjoys the same asymptotic worst-case iteration complexity as TRDH. We report numerical experience on unconstrained and bound-constrained problems, where TRDH and iTRDH are used both as standalone and subproblem solvers. Our results illustrate that, as standalone solvers, TRDH and iTRDH improve upon the quadratic regularization method R2 of <a class="link-https" data-arxiv-id="2103.15993v3" href="https://arxiv.org/abs/2103.15993v3">arXiv:2103.15993v3</a> but also sometimes upon their quasi-Newton trust-region method, referred to here as TR-R2, in terms of smooth objective value and gradient evaluations. On challenging nonnegative matrix factorization, binary classification and data fitting problems, TRDH and iTRDH used as subproblem solvers inside TR improve upon TR-R2 for at least one choice of diagonal approximation.

Pointwise Gradient Estimate of the Ritz Projection

Convergence Analysis of Projected Gradient Descent for Schatten-p Nonconvex Matrix Recovery

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point

An Infeasible-Point Subgradient Method Using Adaptive Approximate Projections

Newton's Method on Riemannian Manifolds: Smale's Point Estimate Theory under the Γ-Condition

Subgradient Projection Algorithms for Convex Feasibility on Riemannian Manifolds with Lower Bounded Curvatures

Riemannian Proximal Gradient Methods (extended Version)

Stability of the Ritz projection in weighted $W^{1,1}$

An Inexact Riemannian Proximal Gradient Method

On the linear convergence rate of Riemannian proximal gradient method

Riemannian proximal gradient methods

Convergence Rate Analysis of a Dykstra-Type Projection Algorithm

The Indefinite Proximal Gradient Method

Riemannian Optimization on Tensor Products of Grassmann Manifolds: Applications to Generalized Rayleigh-Quotients

On the Convergence of Ritz Pairs and Refined Ritz Vectors for Quadratic Eigenvalue Problems

Critical Exponents of the Riesz Projection

An extragradient projection method for strongly quasiconvex equilibrium problems with applications

Weighted Least Approximation on Compact Riemannian Manifolds

Numerical Approximation of Riesz-Feller Operators on $\mathbb R$

A Riemannian Gradient Sampling Algorithm for Nonsmooth Optimization on Manifolds

Convergence of Q-Ritz Pairs, Refined Q-Ritz Vectors and Q-Rayleigh-ritz Method for Quadratic Eigenvalue Problems