Riemannian Bilevel Optimization

Jiaxiang Li,Shiqian Ma

2024-02-08

Abstract:In this work, we consider the bilevel optimization problem on Riemannian manifolds. We inspect the calculation of the hypergradient of such problems on general manifolds and thus enable the utilization of gradient-based algorithms to solve such problems. The calculation of the hypergradient requires utilizing the notion of Riemannian cross-derivative and we inspect the properties and the numerical calculations of Riemannian cross-derivatives. Algorithms in both deterministic and stochastic settings, named respectively RieBO and RieSBO, are proposed that include the existing Euclidean bilevel optimization algorithms as special cases. Numerical experiments on robust optimization on Riemannian manifolds are presented to show the applicability and efficiency of the proposed methods.

Optimization and Control

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper primarily explores bilevel optimization problems on Riemannian manifolds. Specifically:

1. **Hypergradient Computation in Bilevel Optimization Problems**:
- Investigates how to compute the hypergradient in bilevel optimization problems on general Riemannian manifolds.
- Utilizes the concept of Riemannian cross-derivative to compute the hypergradient and analyzes its properties and numerical computation methods.

2. **Algorithm Proposal and Analysis**:
- Proposes two algorithms: a deterministic algorithm RieBO and a stochastic algorithm RieSBO.
- These algorithms include existing bilevel optimization algorithms in Euclidean space as special cases and demonstrate their applicability and efficiency on Riemannian manifolds.
- For deterministic problems (1.1), similar results to those in Euclidean space are obtained through multi-step inner loops and single-step outer loops.
- For stochastic problems (1.2), the framework proposed by Chen et al. (2021b) is adopted, allowing for smaller batch sizes and thus improving algorithm efficiency.

3. **Numerical Experiment Validation**:
- Conducts numerical experiments on distributionally robust optimization problems on Riemannian manifolds, including robust maximum likelihood estimation and the robust Karcher mean problem on the manifold of positive definite matrices.
- Experimental results demonstrate the effectiveness and potential application value of the proposed methods.

In summary, this paper aims to address bilevel optimization problems on Riemannian manifolds and proposes new algorithms and their theoretical analysis suitable for such problems.