Abstract:Many learning tasks are modeled as optimization problems with nonlinear constraints, such as principal component analysis and fitting a Gaussian mixture model. A popular way to solve such problems is resorting to Riemannian optimization algorithms, which yet heavily rely on both human involvement and expert knowledge about Riemannian manifolds. In this paper, we propose a Riemannian meta-optimization method to automatically learn a Riemannian optimizer. We parameterize the Riemannian optimizer by a novel recurrent network and utilize Riemannian operations to ensure that our method is faithful to the geometry of manifolds. The proposed method explores the distribution of the underlying data by minimizing the objective of updated parameters, and hence is capable of learning task-specific optimizations. We introduce a Riemannian implicit differentiation training scheme to achieve efficient training in terms of numerical stability and computational cost. Unlike conventional meta-optimization training schemes that need to differentiate through the whole optimization trajectory, our training scheme is only related to the final two optimization steps. In this way, our training scheme avoids the exploding gradient problem, and significantly reduces the computational load and memory footprint. We discuss experimental results across various constrained problems, including principal component analysis on Grassmann manifolds, face recognition, person re-identification, and texture image classification on Stiefel manifolds, clustering and similarity learning on symmetric positive definite manifolds, and few-shot learning on hyperbolic manifolds.

Riemannian Optimization Methods for the Truncated Takagi Factorization

A Riemannian optimization method on the indefinite Stiefel manifold

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Riemannian optimization on the symplectic Stiefel manifold using second-order information

Riemannian Optimization on Tensor Products of Grassmann Manifolds: Applications to Generalized Rayleigh-Quotients

Riemannian optimization with a preconditioning scheme on the generalized Stiefel manifold

An Extension of Fast Iterative Shrinkage‐thresholding Algorithm to Riemannian Optimization for Sparse Principal Component Analysis

An Extension of Fast Iterative Shrinkage-thresholding to Riemannian Optimization for Sparse Principal Component Analysis

Optimization on product manifolds under a preconditioned metric

Preconditioned low-rank Riemannian optimization for linear systems with tensor product structure

Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

Symplectic Stiefel manifold: tractable metrics, second-order geometry and Newton's methods

Learning to Optimize on Riemannian Manifolds

Convergence and worst-case complexity of adaptive Riemannian trust-region methods for optimization on manifolds

A New Retraction For Accelerating The Riemannian Three-Factor Low-Rank Matrix Completion Algorithm

Low-tubal-rank tensor factorization on constant curvature Riemann manifold with mixture of Gaussians

Nonnegative Tensor Factorizations Using an Alternating Direction Method

Trace Ratio Optimization with an Application to Multi-view Learning

A Riemannian Alternating Descent Ascent Algorithmic Framework for Nonconvex-Linear Minimax Problems on Riemannian Manifolds

Riemannian Optimization Approach to Structure-Preserving Model Order Reduction of Integral-Differential Systems on the Product of Two Stiefel Manifolds

Riemannian Gradient Algorithm for the Numerical Solution of Linear Matrix Equations