Abstract:Many learning tasks are modeled as optimization problems with nonlinear constraints, such as principal component analysis and fitting a Gaussian mixture model. A popular way to solve such problems is resorting to Riemannian optimization algorithms, which yet heavily rely on both human involvement and expert knowledge about Riemannian manifolds. In this paper, we propose a Riemannian meta-optimization method to automatically learn a Riemannian optimizer. We parameterize the Riemannian optimizer by a novel recurrent network and utilize Riemannian operations to ensure that our method is faithful to the geometry of manifolds. The proposed method explores the distribution of the underlying data by minimizing the objective of updated parameters, and hence is capable of learning task-specific optimizations. We introduce a Riemannian implicit differentiation training scheme to achieve efficient training in terms of numerical stability and computational cost. Unlike conventional meta-optimization training schemes that need to differentiate through the whole optimization trajectory, our training scheme is only related to the final two optimization steps. In this way, our training scheme avoids the exploding gradient problem, and significantly reduces the computational load and memory footprint. We discuss experimental results across various constrained problems, including principal component analysis on Grassmann manifolds, face recognition, person re-identification, and texture image classification on Stiefel manifolds, clustering and similarity learning on symmetric positive definite manifolds, and few-shot learning on hyperbolic manifolds.

Two efficient nonlinear conjugate gradient methods for Riemannian manifolds

The convergence of conjugate gradient method with nonmonotone line search

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Two sufficient descent spectral conjugate gradient algorithms for unconstrained optimization with application

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Another Modified Version of RMIL Conjugate Gradient Method

Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

An improved Riemannian conjugate gradient method and its application to robust matrix completion

Optimality conditions for the nonlinear programming problems on Riemannian manifolds

A Riemannian Proximal Newton-CG Method

New vector transport operators extending a Riemannian CG algorithm to generalized Stiefel manifold with low-rank applications

Vector Transport Free Riemannian LBFGS for Optimization on Symmetric Positive Definite Matrix Manifolds

A revised Liu–Storey conjugate gradient parameter for unconstrained optimization problems with applications

Riemannian Proximal Gradient Methods (extended Version)

Riemannian proximal gradient methods

A decent three term conjugate gradient method with global convergence properties for large scale unconstrained optimization problems

Learning to Optimize on Riemannian Manifolds

Global convergence of Riemannian line search methods with a Zhang-Hager-type condition

A family of three-term conjugate gradient projection methods with a restart procedure and their relaxed-inertial extensions for the constrained nonlinear pseudo-monotone equations with applications

Riemannian Anderson Mixing Methods for Minimizing $C^2$-Functions on Riemannian Manifolds