Abstract:The problem of estimating sparse eigenvectors of a symmetric matrix has attracted a lot of attention in many applications, especially those with a high dimensional dataset. While classical eigenvectors can be obtained as the solution of a maximization problem, existing approaches formulate this problem by adding a penalty term into the objective function that encourages a sparse solution. However, the vast majority of the resulting methods achieve sparsity at the expense of sacrificing the orthogonality property. In this paper, we develop a new method to estimate dominant sparse eigenvectors without trading off their orthogonality. The problem is highly nonconvex and hard to handle. We apply the minorizationmaximization framework, wherein we iteratively maximize a tight lower bound (surrogate function) of the objective function over the Stiefel manifold. The inner maximization problem turns out to be a rectangular Procrustes problem, which has a closed-form solution. In addition, we propose a method to improve the covariance estimation problem when its underlying eigenvectors are known to be sparse. We use the eigenvalue decomposition of the covariance matrix to formulate an optimization problem wherein we impose sparsity on the corresponding eigenvectors. Numerical experiments show that the proposed eigenvector extraction algorithm outperforms existing algorithms in terms of support recovery and explained variance, whereas the covariance estimation algorithms improve the sample covariance estimator significantly.

Orthogonal orthogonal overcomplete kernel design for sparse representation

On Sparse Recovery Algorithms in Unions of Orthonormal Bases

A Kaczmarz-Inspired Method for Orthogonalization

Orthogonal Projective Sparse Coding for Image Representation

Orthogonal projection analysis

Approximate Orthogonal Sparse Embedding for Dimensionality Reduction

A Feasible Method for Optimization with Orthogonality Constraints

Online Kernel Learning with a Near Optimal Sparsity Bound

Orthogonal Polynomial Expansions on Sparse Grids.

Complete Dictionary Learning Via L4-Norm Maximization over the Orthogonal Group

Efficient coordinate-descent for orthogonal matrices through Givens rotations

Orthogonal Sparse PCA and Covariance Estimation via Procrustes Reformulation

Orthogonal Procrustes Analysis for Dictionary Learning in Sparse Linear Representation

Analyzing Vector Orthogonalization Algorithms

Sparse principal component analysis by choice of norm

An Exact Penalty Approach for Optimization with Nonnegative Orthogonality Constraints

Complete Dictionary Learning via $\ell^4$-Norm Maximization over the Orthogonal Group

Sparse Optimization for Second Order Kernel Methods

Iterative Orthogonal Matching Pursuit and Sparse Solution

Image feature selection based on orthogonal l2,0 norms