Abstract:This work puts forth low-complexity Riemannian subspace descent algorithms for the minimization of functions over the symmetric positive definite (SPD) manifold. Different from the existing Riemannian gradient descent variants, the proposed approach utilizes carefully chosen subspaces that allow the update to be written as a product of the Cholesky factor of the iterate and a sparse matrix. The resulting updates avoid the costly matrix operations like matrix exponentiation and dense matrix multiplication, which are generally required in almost all other Riemannian optimization algorithms on SPD manifold. We further identify a broad class of functions, arising in diverse applications, such as kernel matrix learning, covariance estimation of Gaussian distributions, maximum likelihood parameter estimation of elliptically contoured distributions, and parameter estimation in Gaussian mixture model problems, over which the Riemannian gradients can be calculated efficiently. The proposed uni-directional and multi-directional Riemannian subspace descent variants incur per-iteration complexities of $O(n)$ and $O(n^2)$ respectively, as compared to the $O(n^3)$ or higher complexity incurred by all existing Riemannian gradient descent variants. The superior runtime and low per-iteration complexity of the proposed algorithms is also demonstrated via numerical tests on large-scale covariance estimation and matrix square root problems. MATLAB code implementation is publicly available on GitHub : <a class="link-external link-https" href="https://github.com/yogeshd-iitk/subspace_descent_over_SPD_manifold" rel="external noopener nofollow">this https URL</a>

An Iterative Coordinate Descent Algorithm to Compute Sparse Low-Rank Approximations

Row-Sparse Principal Component Analysis via Coordinate Descent Method

Learning-Based Low-Rank Approximations

A Coordinate-wise Optimization Algorithm for Sparse Inverse Covariance Selection.

Krylov Subspace Recycling for Fast Iterative Least-Squares in Machine Learning

Orthogonal Sparse PCA and Covariance Estimation via Procrustes Reformulation

Low-Rank Approximations with Sparse Factors I: Basic Algorithms and Error Analysis

Gradient-based sparse principal component analysis with extensions to online learning

An Efficient Gauss--Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations

A Covariance-Free Iterative Principal Component Analysis for High Dimensional and Large Scale Data

Efficient coordinate-descent for orthogonal matrices through Givens rotations

Sparse principal component analysis via regularized low rank matrix approximation

An algorithm for low-rank matrix factorization and its applications

Greedy Double Subspaces Coordinate Descent Method for Solving Linear Least-Squares Problems

Sparse Unsupervised Dimensionality Reduction Algorithms

Generalized Low-rank plus Sparse Tensor Estimation by Fast Riemannian Optimization

Learning Feature Sparse Principal Subspace.

A non-convex approach to low-rank and sparse matrix decomposition.

Low-complexity subspace-descent over symmetric positive definite manifold

Sparse principal component of a rank-deficient matrix

Implementation improvements and extensions of an ODE-based algorithm for structured low-rank approximation