Abstract:Many learning tasks are modeled as optimization problems with nonlinear constraints, such as principal component analysis and fitting a Gaussian mixture model. A popular way to solve such problems is resorting to Riemannian optimization algorithms, which yet heavily rely on both human involvement and expert knowledge about Riemannian manifolds. In this paper, we propose a Riemannian meta-optimization method to automatically learn a Riemannian optimizer. We parameterize the Riemannian optimizer by a novel recurrent network and utilize Riemannian operations to ensure that our method is faithful to the geometry of manifolds. The proposed method explores the distribution of the underlying data by minimizing the objective of updated parameters, and hence is capable of learning task-specific optimizations. We introduce a Riemannian implicit differentiation training scheme to achieve efficient training in terms of numerical stability and computational cost. Unlike conventional meta-optimization training schemes that need to differentiate through the whole optimization trajectory, our training scheme is only related to the final two optimization steps. In this way, our training scheme avoids the exploding gradient problem, and significantly reduces the computational load and memory footprint. We discuss experimental results across various constrained problems, including principal component analysis on Grassmann manifolds, face recognition, person re-identification, and texture image classification on Stiefel manifolds, clustering and similarity learning on symmetric positive definite manifolds, and few-shot learning on hyperbolic manifolds.

Nonsmooth nonconvex optimization on Riemannian manifolds via bundle trust region algorithm

Riemannian Trust Region Methods for Minimization

The Riemannian Convex Bundle Method

A Trust Region-Type Method for Solving Monotone Variational Inequality

A TRUST-REGION METHOD FOR NONSMOOTH NONCONVEX OPTIMIZATION

Convergence and worst-case complexity of adaptive Riemannian trust-region methods for optimization on manifolds

Riemannian Trust Region Methods for SC$^1$ Minimization

A restricted memory quasi-Newton bundle method for nonsmooth optimization on Riemannian manifolds

A trust region-type normal map-based semismooth Newton method for nonsmooth nonconvex composite optimization

Riemannian trust-region methods for strict saddle functions with complexity guarantees

A Riemannian Gradient Sampling Algorithm for Nonsmooth Optimization on Manifolds

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

A Riemannian optimization method on the indefinite Stiefel manifold

A Riemannian Subspace BFGS Trust Region Method

An interior-point trust-region method for nonsmooth regularized bound-constrained optimization

A Trust Region Proximal Gradient Method for Nonlinear Multi-objective Optimization Problems

Riemannian optimization on the symplectic Stiefel manifold using second-order information

Vector Transport Free Riemannian LBFGS for Optimization on Symmetric Positive Definite Matrix Manifolds

A new method based on the bundle idea and gradient sampling technique for minimizing nonsmooth convex functions

Learning to Optimize on Riemannian Manifolds

A new trust region algorithm for nonsmooth convex minimization