Abstract:Optimization on the Stiefel manifold or with orthogonality constraints is an important problem in many signal processing and data analysis applications such as Sparse Principal Component Analysis (SPCA). Algorithms such as the Riemannian proximal gradient algorithms addressing this problem usually involve an intricate subproblem requiring a semi-smooth Newton method hence, simple and effective operator splitting methods extended to the manifold setting such as the Alternating Direction Method of Multipliers (ADMM) have been proposed. However, another simple operator-splitting method, the Quadratic Penalty Alternating Minimization (QPAM) method which has been successful in image processing to our knowledge, has not yet been extended to the manifold setting. In this paper, we propose a manifold QPAM (MQPAM) which is very simple to implement. The iterative scheme of the MQPAM consists of a Riemannian Gradient Descent (RGD) subproblem and a subproblem in the form of a proximal operator which has a closed-form solution. Experiments on the SPCA problem show that our proposed MQPAM is at par with or better than several other algorithms in terms of sparsity and CPU time.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the optimization problem in Sparse Principal Component Analysis (SPCA), especially non - smooth non - convex Riemannian optimization on the Stiefel manifold. Specifically, SPCA aims to find a low - dimensional representation from high - dimensional datasets and improve the interpretability of data by imposing sparsity and orthogonality constraints. ### Problem Background 1. **Sparse Principal Component Analysis (SPCA)** - PCA is a commonly used data analysis tool for projecting high - dimensional data into a low - dimensional space. However, the loading vectors obtained by traditional PCA are usually dense, which can lead to difficult - to - interpret results in some application scenarios. - To improve the interpretability of data, SPCA makes the loading vectors sparser by introducing sparsity constraints (such as ℓ₁ - norm regularization) and orthogonality constraints (such as Stiefel manifold constraints). 2. **Form of the Optimization Problem** - The form focused on in this paper is: \[ \min_{X} f(X)+g(X)\quad\text{subject to}\quad X\in M \] where \(M = St(n, p)=\{X\in\mathbb{R}^{n\times p}:X^{\top}X = I_p\}\) is the Stiefel manifold, \(f\) is a smooth but possibly non - convex function, and \(g\) is a non - smooth and convex function. 3. **Limitations of Existing Methods** - Existing Riemannian optimization algorithms (such as the Riemannian proximal gradient method) usually involve complex sub - problem solving, such as the semismooth Newton method, which increases the computation time. - Some algorithms customize the dual - variable update steps (such as Lagrange multiplier updates), further increasing the complexity. ### Method Proposed in the Paper The paper proposes a new method - **Manifold Quadratic Penalty Alternating Minimization (MQPAM)** to simplify the solution process of the SPCA problem. The main features of MQPAM include: - **Simple and Easy to Implement**: MQPAM only requires one Riemannian gradient descent (RGD) step and a proximal operation with a closed - form solution. - **Efficient and Fast**: Since it does not involve complex inner iterations or dual - variable updates, MQPAM performs well in CPU time. - **Related to Moreau Smoothing Technique**: The Y - subproblem can be interpreted by the Moreau envelope function, thus simplifying the calculation. ### Experimental Results The experimental results show that MQPAM performs excellently in CPU time and sparsity. Especially under smaller regularization parameters, it can obtain better sparsity results. ### Summary The paper proposes a novel MQPAM algorithm specifically for solving the SPCA problem on the Stiefel manifold. This method not only achieves efficient optimization but also maintains high sparsity, providing a new solution for sparse principal component analysis.

Manifold Quadratic Penalty Alternating Minimization for Sparse Principal Component Analysis

Nonlinear Eigen-approach ADMM for Sparse Optimization on Stiefel Manifold

An Alternating Manifold Proximal Gradient Method for Sparse Principal Component Analysis and Sparse Canonical Correlation Analysis

Learning Sparse PCA with Stabilized ADMM Method on Stiefel Manifold

An Alternating Manifold Proximal Gradient Method for Sparse PCA and Sparse CCA

An Extension of Fast Iterative Shrinkage‐thresholding Algorithm to Riemannian Optimization for Sparse Principal Component Analysis

An Extension of Fast Iterative Shrinkage-thresholding to Riemannian Optimization for Sparse Principal Component Analysis

Riemannian Proximal Stochastic Gradient Descent for Sparse 2DPCA.

A Riemannian Alternating Descent Ascent Algorithmic Framework for Nonconvex-Linear Minimax Problems on Riemannian Manifolds

An Efficient Algorithm for Tensor Principal Component Analysis Via Proximal Linearized Alternating Direction Method of Multipliers

An Alternating Minimization Method for Robust Principal Component Analysis

A Customized Inertial Proximal Alternating Minimization for SVD-free Robust Principal Component Analysis

Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation

Low-complexity subspace-descent over symmetric positive definite manifold

Proximal methods for structured nonsmooth optimization over Riemannian submanifolds

Alternating Direction Method of Multipliers for Sparse Principal Component Analysis

An Inertial Proximal Partially Symmetric ADMM-based Algorithm for Linearly Constrained Multi-Block Nonconvex Optimization Problems with Applications.

An Efficient Alternating Riemannian/Projected Gradient Descent Ascent Algorithm for Fair Principal Component Analysis

Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation.

A Proximal Peaceman-Rachford Splitting Method for Solving the Multi-Block Separable Convex Minimization Problems.