Abstract:In machine learning research, the proximal gradient methods are popular for solving various optimization problems with non-smooth regularization. Inexact proximal gradient methods are extremely important when exactly solving the proximal operator is time-consuming, or the proximal operator does not have an analytic solution. However, existing inexact proximal gradient methods only consider convex problems. The knowledge of inexact proximal gradient methods in the non-convex setting is very limited. % Moreover, for some machine learning models, there is still no proposed solver for exactly solving the proximal operator. To address this challenge, in this paper, we first propose three inexact proximal gradient algorithms, including the basic version and Nesterov's accelerated version. After that, we provide the theoretical analysis to the basic and Nesterov's accelerated versions. The theoretical results show that our inexact proximal gradient algorithms can have the same convergence rates as the ones of exact proximal gradient algorithms in the non-convex setting. Finally, we show the applications of our inexact proximal gradient algorithms on three representative non-convex learning problems. All experimental results confirm the superiority of our new inexact proximal gradient algorithms.

An Inexact Riemannian Proximal Gradient Method

On the linear convergence rate of Riemannian proximal gradient method

A Riemannian Proximal Newton Method

Inexact Riemannian Gradient Descent Method for Nonconvex Optimization

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point

Riemannian proximal gradient methods

Riemannian Proximal Gradient Methods (extended Version)

Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Proximal methods for structured nonsmooth optimization over Riemannian submanifolds

Nonsmooth Optimization over the Stiefel Manifold and Beyond: Proximal Gradient Method and Recent Variants

Convergence analysis of inexact proximal point algorithms on Hadamard manifolds

Inexact Proximal Gradient Methods for Non-convex and Non-smooth Optimization

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

An inexact regularized proximal Newton method for nonconvex and nonsmooth optimization

A Riemannian Proximal Newton-CG Method

Proximal gradient method for nonsmooth optimization over the Stiefel manifold

The Proximal Gradient Method for Composite Optimization Problems on Riemannian Manifolds

Proximal Gradient Algorithm with Trust Region Scheme on Riemannian Manifold