Abstract:Sparse Principal Component Analysis (Sparse PCA) is a pivotal tool in data analysis and dimensionality reduction. However, Sparse PCA is a challenging problem in both theory and practice: it is known to be NP-hard and current exact methods generally require exponential runtime. In this paper, we propose a novel framework to efficiently approximate Sparse PCA by (i) approximating the general input covariance matrix with a re-sorted block-diagonal matrix, (ii) solving the Sparse PCA sub-problem in each block, and (iii) reconstructing the solution to the original problem. Our framework is simple and powerful: it can leverage any off-the-shelf Sparse PCA algorithm and achieve significant computational speedups, with a minor additive error that is linear in the approximation error of the block-diagonal matrix. Suppose $g(k, d)$ is the runtime of an algorithm (approximately) solving Sparse PCA in dimension $d$ and with sparsity value $k$. Our framework, when integrated with this algorithm, reduces the runtime to $\mathcal{O}\left(\frac{d}{d^\star} \cdot g(k, d^\star) + d^2\right)$, where $d^\star \leq d$ is the largest block size of the block-diagonal matrix. For instance, integrating our framework with the Branch-and-Bound algorithm reduces the complexity from $g(k, d) = \mathcal{O}(k^3\cdot d^k)$ to $\mathcal{O}(k^3\cdot d \cdot (d^\star)^{k-1})$, demonstrating exponential speedups if $d^\star$ is small. We perform large-scale evaluations on many real-world datasets: for exact Sparse PCA algorithm, our method achieves an average speedup factor of 93.77, while maintaining an average approximation error of 2.15%; for approximate Sparse PCA algorithm, our method achieves an average speedup factor of 6.77 and an average approximation error of merely 0.37%.

Distributed Principal Component Analysis Based on Randomized Low-Rank Approximation.

Randomized algorithms for distributed computation of principal component analysis and singular value decomposition

A Communication-Efficient Distributed Algorithm for Kernel Principal Component Analysis.

Robust Covariance Estimation for Distributed Principal Component Analysis

Distributed estimation of principal eigenspaces

A linearly convergent algorithm for distributed principal component analysis

Distributed Estimation for Principal Component Analysis: An Enlarged Eigenspace Analysis

FAST-PCA: A Fast and Exact Algorithm for Distributed Principal Component Analysis

A Covariance-Free Iterative Algorithm for Distributed Principal Component Analysis on Vertically Partitioned Data.

Fast Randomized PCA for Sparse Data

DeEPCA: Decentralized Exact PCA with Linear Convergence Rate

Online Learning over Distributed Low-Rank Networks Via Sequential Power Iteration

Online Distributed Estimation of Principal Eigenspaces

A Distributed Gradient Algorithm Based on Randomized Block-Coordinate and Projection-Free over Networks

Distributed Parallel PCA for Modeling and Monitoring of Large-Scale Plant-Wide Processes with Big Data.

Communication-Efficient Distributed Covariance Sketch, with Application to Distributed PCA

Distributed Tensor Principal Component Analysis

Hyperdimensional Computing for Efficient Distributed Classification with Randomized Neural Networks

Efficient Matrix Sketching over Distributed Data

Efficient Sparse PCA via Block-Diagonalization

A Generalized Mean Approach for Distributed-PCA