A general framework of Riemannian adaptive optimization methods with a convergence analysis

Hiroyuki Sakai,Hideaki Iiduka
2024-09-02
Abstract:This paper proposes a general framework of Riemannian adaptive optimization methods. The framework encapsulates several stochastic optimization algorithms on Riemannian manifolds and incorporates the mini-batch strategy that is often used in deep learning. Within this framework, we also propose AMSGrad on embedded submanifolds of Euclidean space. Moreover, we give convergence analyses valid for both a constant and a diminishing step size. Our analyses also reveal the relationship between the convergence rate and mini-batch size. In numerical experiments, we applied the proposed algorithm to principal component analysis and the low-rank matrix completion problem, which can be considered to be Riemannian optimization problems. Python implementations of the methods used in the numerical experiments are available at <a class="link-external link-https" href="https://github.com/iiduka-researches/202408-adaptive" rel="external noopener nofollow">this https URL</a>.
Optimization and Control
What problem does this paper attempt to address?
The paper aims to address the following issues: 1. **Propose a General Framework**: The paper proposes a general framework (Algorithm 1) for adaptive optimization methods on Riemannian manifolds. This framework can encapsulate various stochastic optimization algorithms and incorporates the mini-batch strategy commonly used in deep learning. 2. **Propose AMSGrad on Embedded Submanifolds**: Within this framework, the authors further propose a variant of AMSGrad on submanifolds embedded in Euclidean space (Algorithm 2). 3. **Convergence Analysis**: The authors conduct a convergence analysis of the proposed algorithms, proving that the method is applicable to both constant step sizes and diminishing step sizes, and revealing the relationship between the convergence rate and the mini-batch size. Through numerical experiments, the authors apply the proposed methods to Principal Component Analysis (PCA) and low-rank matrix completion problems, demonstrating their performance on different datasets. The experimental results indicate that the proposed adaptive optimization methods outperform existing Riemannian adaptive optimization methods in certain cases.