Fast Spectral Low Rank Matrix Approximation

Haishan Ye,Zhihua Zhang

2015-01-01

Abstract:First, we extend the results of approximate matrix multiplication from the Frobenius norm to the spectral norm. Second, We develop a class of fast approximate generalized linear regression algorithms with respect to the spectral norm. Finally, We give a fast approximate SVD.

What problem does this paper attempt to address?

Fast randomized algorithms for low-rank matrix approximations with applications in global comparative analysis of a class of data sets

Weiwei Xu,Weijie Shen,Wen Li,Weiguo Gao,Yingzhou Li

2024-04-15

Abstract:Generalized singular values (GSVs) play an essential role in the comparative analysis. In the real world data for comparative analysis, both data matrices are usually numerically low-rank. This paper proposes a randomized algorithm to first approximately extract bases and then calculate GSVs efficiently. The accuracy of both basis extration and comparative analysis quantities, angular distances, generalized fractions of the eigenexpression, and generalized normalized Shannon entropy, are rigursly analyzed. The proposed algorithm is applied to both synthetic data sets and the genome-scale expression data sets. Comparing to other GSVs algorithms, the proposed algorithm achieves the fastest runtime while preserving sufficient accuracy in comparative analysis.

Numerical Analysis
An algorithm for low-rank matrix factorization and its applications

Baiyu Chen,Zi Yang,Zhouwang Yang

DOI: https://doi.org/10.1016/j.neucom.2017.09.052

IF: 6

2018-01-01

Neurocomputing

Abstract:This paper proposes a valid and fast algorithm for low-rank matrix factorization. There are multiple applications for low-rank matrix factorization, and numerous algorithms have been developed to solve this problem. However, many algorithms do not use rank directly; instead, they minimize a nuclear norm by using Singular Value Decomposition (SVD), which requires a huge time cost. In addition, these algorithms often fix the dimension of the factorized matrix, meaning that one must first find an optimum dimension for the factorized matrix in order to obtain a solution. Unfortunately, the optimum dimension is unknown in many practical problems, such as matrix completion and recommender systems. Therefore, it is necessary to develop a faster algorithm that can also estimate the optimum dimension. In this paper, we use the Hidden Matrix Factorized Augmented Lagrangian Method to solve low-rank matrix factorizations. We also add a tool to dynamically estimate the optimum dimension and adjust it while simultaneously running the algorithm. Additionally, in the era of Big Data, there will be more and more large, sparse data. In face of such highly sparse data, our algorithm has the potential to be more effective than other algorithms. We applied it to some practical problems, e.g. Low-Rank Representation(LRR), and matrix completion with constraint. In numerical experiments, it has performed well when applied to both synthetic data and real-world data.

computer science, artificial intelligence
Fast and Accurate Matrix Completion Via Truncated Nuclear Norm Regularization

Yao Hu,Debing Zhang,Jieping Ye,Xuelong Li,Xiaofei He

DOI: https://doi.org/10.1109/tpami.2012.271

IF: 23.6

2013-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Recovering a large matrix from a small subset of its entries is a challenging problem arising in many real applications, such as image inpainting and recommender systems. Many existing approaches formulate this problem as a general low-rank matrix approximation problem. Since the rank operator is nonconvex and discontinuous, most of the recent theoretical studies use the nuclear norm as a convex relaxation. One major limitation of the existing approaches based on nuclear norm minimization is that all the singular values are simultaneously minimized, and thus the rank may not be well approximated in practice. In this paper, we propose to achieve a better approximation to the rank of matrix by truncated nuclear norm, which is given by the nuclear norm subtracted by the sum of the largest few singular values. In addition, we develop a novel matrix completion algorithm by minimizing the Truncated Nuclear Norm. We further develop three efficient iterative procedures, TNNR-ADMM, TNNR-APGL, and TNNR-ADMMAP, to solve the optimization problem. TNNR-ADMM utilizes the alternating direction method of multipliers (ADMM), while TNNR-AGPL applies the accelerated proximal gradient line search method (APGL) for the final optimization. For TNNR-ADMMAP, we make use of an adaptive penalty according to a novel update rule for ADMM to achieve a faster convergence rate. Our empirical study shows encouraging results of the proposed algorithms in comparison to the state-of-the-art matrix completion algorithms on both synthetic and real visual datasets.
Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares

Trevor Hastie,Rahul Mazumder,Jason Lee,Reza Zadeh

DOI: https://doi.org/10.48550/arXiv.1410.2596

2014-10-09

Methodology

Abstract:The matrix-completion problem has attracted a lot of attention, largely as a result of the celebrated Netflix competition. Two popular approaches for solving the problem are nuclear-norm-regularized matrix approximation (Candes and Tao, 2009, Mazumder, Hastie and Tibshirani, 2010), and maximum-margin matrix factorization (Srebro, Rennie and Jaakkola, 2005). These two procedures are in some cases solving equivalent problems, but with quite different algorithms. In this article we bring the two approaches together, leading to an efficient algorithm for large matrix factorization and completion that outperforms both of these. We develop a software package "softImpute" in R for implementing our approaches, and a distributed version for very large matrices using the "Spark" cluster programming environment.
Fast Spectrum Estimation of Some Kernel Matrices

Mikhail Lepilov

2024-11-01

Abstract:In data science, individual observations are often assumed to come independently from an underlying probability space. Kernel matrices formed from large sets of such observations arise frequently, for example during classification tasks. It is desirable to know the eigenvalue decay properties of these matrices without explicitly forming them, such as when determining if a low-rank approximation is feasible. In this work, we introduce a new eigenvalue quantile estimation framework for some kernel matrices. This framework gives meaningful bounds for all the eigenvalues of a kernel matrix while avoiding the cost of constructing the full matrix. The kernel matrices under consideration come from a kernel with quick decay away from the diagonal applied to uniformly-distributed sets of points in Euclidean space of any dimension. We prove the efficacy of this framework given certain bounds on the kernel function, and we provide empirical evidence for its accuracy. In the process, we also prove a very general interlacing-type theorem for finite sets of numbers. Additionally, we indicate an application of this framework to the study of the intrinsic dimension of data, as well as several other directions in which to generalize this work.

Machine Learning,Numerical Analysis
Low-Rank Approximations with Sparse Factors I: Basic Algorithms and Error Analysis

ZY Zhang,HY Zha,H Simon

DOI: https://doi.org/10.1137/s0895479899359631

IF: 1.908

2002-01-01

SIAM Journal on Matrix Analysis and Applications

Abstract:We consider the problem of computing low-rank approximations of matrices. The novel aspects of our approach are that we require the low-rank approximations to be written in a factorized form with sparse factors, and the degree of sparsity of the factors can be traded off for reduced reconstruction error by certain user-determined parameters. We give a detailed error analysis of our proposed algorithms and compare the computed sparse low-rank approximations with those obtained from singular value decomposition. We present numerical examples arising from some application areas to illustrate the efficiency and accuracy of our algorithms.
Fast Low-Rank Matrix Learning with Nonconvex Regularization

Quanming Yao,James T. Kwok,Wenliang Zhong

DOI: https://doi.org/10.1109/icdm.2015.9

IF: 23.6

2019-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Low-rank modeling has a lot of important applications in machine learning, computer vision and social network analysis. While the matrix rank is often approximated by the convex nuclear norm, the use of nonconvex low-rank regularizers has demonstrated better recovery performance. However, the resultant optimization problem is much more challenging. A very recent state-of-the-art is based on the proximal gradient algorithm. However, it requires an expensive full SVD in each proximal step. In this paper, we show that for many commonly-used nonconvex low-rank regularizers, a cutoff can be derived to automatically threshold the singular values obtained from the proximal operator. This allows the use of power method to approximate the SVD efficiently. Besides, the proximal operator can be reduced to that of a much smaller matrix projected onto this leading subspace. Convergence, with a rate of O(1/T) where T is the number of iterations, can be guaranteed. Extensive experiments are performed on matrix completion and robust principal component analysis. The proposed method achieves significant speedup over the state-of-the-art. Moreover, the matrix solution obtained is more accurate and has a lower rank than that of the traditional nuclear norm regularizer.
Fast $(1+\varepsilon)$-Approximation Algorithms for Binary Matrix Factorization

Ameya Velingker,Maximilian Vötsch,David P. Woodruff,Samson Zhou

2023-06-03

Abstract:We introduce efficient $(1+\varepsilon)$-approximation algorithms for the binary matrix factorization (BMF) problem, where the inputs are a matrix $\mathbf{A}\in\{0,1\}^{n\times d}$, a rank parameter $k>0$, as well as an accuracy parameter $\varepsilon>0$, and the goal is to approximate $\mathbf{A}$ as a product of low-rank factors $\mathbf{U}\in\{0,1\}^{n\times k}$ and $\mathbf{V}\in\{0,1\}^{k\times d}$. Equivalently, we want to find $\mathbf{U}$ and $\mathbf{V}$ that minimize the Frobenius loss $\|\mathbf{U}\mathbf{V} - \mathbf{A}\|_F^2$. Before this work, the state-of-the-art for this problem was the approximation algorithm of Kumar et. al. [ICML 2019], which achieves a $C$-approximation for some constant $C\ge 576$. We give the first $(1+\varepsilon)$-approximation algorithm using running time singly exponential in $k$, where $k$ is typically a small integer. Our techniques generalize to other common variants of the BMF problem, admitting bicriteria $(1+\varepsilon)$-approximation algorithms for $L_p$ loss functions and the setting where matrix operations are performed in $\mathbb{F}_2$. Our approach can be implemented in standard big data models, such as the streaming or distributed models.

Data Structures and Algorithms,Machine Learning
Efficient Randomized Algorithms for the Fixed-Precision Low-Rank Matrix Approximation

Wenjian Yu,Yu Gu,Yaohang Li

DOI: https://doi.org/10.1137/17M1141977

IF: 1.908

2019-10-07

SIAM Journal on Matrix Analysis and Applications

Abstract:Randomized algorithms for low-rank matrix approximation are investigated, with the emphasis on the fixed-precision problem and computational efficiency for handling large matrices. The algorithms are based on the so-called QB factorization, where Q is an orthonormal matrix. First, a mechanism for calculating the approximation error in the Frobenius norm is proposed, which enables efficient adaptive rank determination for a large and/or sparse matrix. It can be combined with any QB-form factorization algorithm in which B's rows are incrementally generated. Based on the blocked randQB algorithm by Martinsson and Voronin, this results in an algorithm called randQB_EI. Then, we further revise the algorithm to obtain a pass-efficient algorithm, randQB_FP, which is mathematically equivalent to the existing randQB algorithms and also suitable for the fixed-precision problem. Especially, randQB_FP can serve as a single-pass algorithm for calculating leading singular values, under a certain condition. With large and/or sparse test matrices, we have empirically validated the merits of the proposed techniques, which exhibit remarkable speedup and memory saving over the blocked randQB algorithm. We have also demonstrated that the single-pass algorithm derived by randQB_FP is much more accurate than an existing single-pass algorithm. And with data from a scenic image and an information retrieval application, we have shown the advantages of the proposed algorithms over the adaptive range finder algorithm for solving the fixed-precision problem.

mathematics, applied
An Efficient Gauss--Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations

Xin Liu,Zaiwen Wen,Yin Zhang

DOI: https://doi.org/10.1137/140971464

IF: 2.763

2015-01-01

SIAM Journal on Optimization

Abstract:We derive and study a Gauss-Newton method for computing a symmetric low-rank product XXT, where X is an element of R-nxk for k < n, that is the closest to a given symmetric matrix A is an element of R-nxn in Frobenius norm. When A = (BB)-B-T (or BBT), this problem essentially reduces to finding a truncated singular value decomposition of B. Our Gauss-Newton method, which has a particularly simple form, shares the same order of iteration-complexity as a gradient method when k << n, but can be significantly faster on a wide range of problems. In this paper, we prove global convergence and a Q-linear convergence rate for this algorithm and perform numerical experiments on various test problems, including those from recently active areas of matrix completion and robust principal component analysis. Numerical results show that the proposed algorithm is capable of providing considerable speed advantages over Krylov subspace methods on suitable application problems where high-accuracy solutions are not required. Moreover, the algorithm possesses a higher degree of concurrency than Krylov subspace methods, thus offering better scalability on modern multi-/many-core computers.
Fast and Low Memory Cost Matrix Factorization: Algorithm, Analysis, and Case Study.

Yan,Mingkui Tan,Ivor W. Tsang,Yi Yang,Qinfeng Shi,Chengqi Zhang

DOI: https://doi.org/10.1109/tkde.2018.2882197

IF: 9.235

2018-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Matrix factorization has been widely applied to various applications. With the fast development of storage and internet technologies, we have been witnessing a rapid increase of data. In this paper, we propose new algorithms for matrix factorization with the emphasis on efficiency. In addition, most existing methods of matrix factorization only consider a general smooth least square loss. Differently, many real-world applications have distinctive characteristics. As a result, different losses should be used accordingly. Therefore, it is beneficial to design new matrix factorization algorithms that are able to deal with both smooth and non-smooth losses. To this end, one needs to analyze the characteristics of target data and use the most appropriate loss based on the analysis. We particularly study two representative cases of low-rank matrix recovery, i.e., collaborative filtering for recommendation and high dynamic range imaging. To solve these two problems, we respectively propose a stage-wise matrix factorization algorithm by exploiting manifold optimization techniques. From our theoretical analysis, they are both are provably guaranteed to converge to a stationary point. Extensive experiments on recommender systems and high dynamic range imaging demonstrate the satisfactory performance and efficiency of our proposed method on large-scale real data.
Fast Machine-Precision Spectral Likelihoods for Stationary Time Series

Christopher J. Geoga

2024-05-11

Abstract:We provide in this work an algorithm for approximating a very broad class of symmetric Toeplitz matrices to machine precision in $\mathcal{O}(n \log n)$ time with applications to fitting time series models. In particular, for a symmetric Toeplitz matrix $\mathbf{\Sigma}$ with values $\mathbf{\Sigma}_{j,k} = h_{|j-k|} = \int_{-1/2}^{1/2} e^{2 \pi i |j-k| \omega} S(\omega) \mathrm{d} \omega$ where $S(\omega)$ is piecewise smooth, we give an approximation $\mathbf{\mathcal{F}} \mathbf{\Sigma} \mathbf{\mathcal{F}}^H \approx \mathbf{D} + \mathbf{U} \mathbf{V}^H$, where $\mathbf{\mathcal{F}}$ is the DFT matrix, $\mathbf{D}$ is diagonal, and the matrices $\mathbf{U}$ and $\mathbf{V}$ are in $\mathbb{C}^{n \times r}$ with $r \ll n$. Studying these matrices in the context of time series, we offer a theoretical explanation of this structure and connect it to existing spectral-domain approximation frameworks. We then give a complete discussion of the numerical method for assembling the approximation and demonstrate its efficiency for improving Whittle-type likelihood approximations, including dramatic examples where a correction of rank $r = 2$ to the standard Whittle approximation increases the accuracy from $3$ to $14$ digits for a matrix $\mathbf{\Sigma} \in \mathbb{R}^{10^5 \times 10^5}$. The method and analysis of this work applies well beyond time series analysis, providing an algorithm for extremely accurate direct solves with a wide variety of symmetric Toeplitz matrices. The analysis employed here largely depends on asymptotic expansions of oscillatory integrals, and also provides a new perspective on when existing spectral-domain approximation methods for Gaussian log-likelihoods can be particularly problematic.

Numerical Analysis,Computation,Methodology
Fast Generalized Matrix Regression with Applications in Machine Learning

Ye Haishan,Wang Shusen,Zhang Zhihua,Zhang Tong

2019-01-01

Abstract: Fast matrix algorithms have become the fundamental tools of machine learning in big data era. The generalized matrix regression problem is widely used in the matrix approximation such as CUR decomposition, kernel matrix approximation, and stream singular value decomposition (SVD), etc. In this paper, we propose a fast generalized matrix regression algorithm (Fast GMR) which utilizes sketching technique to solve the GMR problem efficiently. Given error parameter $0<\epsilon<1$, the Fast GMR algorithm can achieve a $(1+\epsilon)$ relative error with the sketching sizes being of order $\cO(\epsilon^{-1/2})$ for a large group of GMR problems. We apply the Fast GMR algorithm to the symmetric positive definite matrix approximation and single pass singular value decomposition and they achieve a better performance than conventional algorithms. Our empirical study also validates the effectiveness and efficiency of our proposed algorithms.
A Fast Spectral Algorithm for Mean Estimation with Sub-Gaussian Rates

Zhixian Lei,Kyle Luh,Prayaag Venkat,Fred Zhang

2020-02-18

Abstract:We study the algorithmic problem of estimating the mean of heavy-tailed random vector in $\mathbb{R}^d$, given $n$ i.i.d. samples. The goal is to design an efficient estimator that attains the optimal sub-gaussian error bound, only assuming that the random vector has bounded mean and covariance. Polynomial-time solutions to this problem are known but have high runtime due to their use of semi-definite programming (SDP). Conceptually, it remains open whether convex relaxation is truly necessary for this problem.

Machine Learning,Data Structures and Algorithms
Accurate Low Rank Approximation at a Low Computational Cost

Soo Go,Qi Luan,Victor Y. Pan

2024-05-09

Abstract:At sublinear computational cost we greatly decrease the spectral error norm of a crude Low Rank Approximation LRA) of a matrix that has fast decaying spectrum of singular values. Our formal estimates for this decrease is in good accordance with our numerical tests for synthetic and real-world inputs. Moreover, empirically, for a large subclass of inputs with fast decaying spectra, the entire LRA with our escalation was accurate and had sublinear cost.

Numerical Analysis
A generalized Nystrom method with column sketching for low-rank approximation of nonsymmetric matrices

Yatian Wang,Hua Xiang,Chi Zhang,Songling Zhang

2024-10-29

Abstract:This paper is concerned with the low-rank approximation for large-scale nonsymmetric matrices. Inspired by the classical Nystrom method, which is a popular method to find the low-rank approximation for symmetric positive semidefinite matrices, we explore an extension of the Nystrom method to approximate nonsymmetric matrices. The proposed method is a generalized Nystrom method with column sketching and shows its advantages in accuracy and speed without sacri cing stability. And the numerical experiments will illustrate the robustness of our new methods in finding a desired low-rank approximation of nonsymmetric matrix.

Numerical Analysis
Efficient direct position determination using Frobenius norm approximation

Xuefeng Feng,Zhen Huang,Jiazhi He,Dingli Xue

DOI: https://doi.org/10.1049/ell2.12479

2022-01-01

Electronics Letters

Abstract:To eliminate the bottleneck of computation and communication capability in classic direct position determination, recently several fast approximation methods have been proposed. However, the estimation accuracy of these methods is not satisfactory. In this letter, the common theoretical principle behind these fast approximation methods is analysed in depth. It is demonstrated that the maximum eigenvalue of a matrix can be approximated by the matrix norm. The performance of several commonly used matrix norms is then compared, and the Frobenius norm is preferable because of its great potential inferred from the analysis in this letter. Numerical simulations indicate that, compared with conventional methods, the proposed method can achieve better estimation accuracy with less computational complexity.
Sublinear Time Spectral Density Estimation

Vladimir Braverman,Aditya Krishnan,Christopher Musco

DOI: https://doi.org/10.1145/3519935.3520009

2022-04-15

Abstract:We present a new sublinear time algorithm for approximating the spectral density (eigenvalue distribution) of an $n\times n$ normalized graph adjacency or Laplacian matrix. The algorithm recovers the spectrum up to $\epsilon$ accuracy in the Wasserstein-1 distance in $O(n\cdot \text{poly}(1/\epsilon))$ time given sample access to the graph. This result compliments recent work by David Cohen-Steiner, Weihao Kong, Christian Sohler, and Gregory Valiant (2018), which obtains a solution with runtime independent of $n$, but exponential in $1/\epsilon$. We conjecture that the trade-off between dimension dependence and accuracy is inherent. Our method is simple and works well experimentally. It is based on a Chebyshev polynomial moment matching method that employees randomized estimators for the matrix trace. We prove that, for any Hermitian $A$, this moment matching method returns an $\epsilon$ approximation to the spectral density using just $O({1}/{\epsilon})$ matrix-vector products with $A$. By leveraging stability properties of the Chebyshev polynomial three-term recurrence, we then prove that the method is amenable to the use of coarse approximate matrix-vector products. Our sublinear time algorithm follows from combining this result with a novel sampling algorithm for approximating matrix-vector products with a normalized graph adjacency matrix. Of independent interest, we show a similar result for the widely used \emph{kernel polynomial method} (KPM), proving that this practical algorithm nearly matches the theoretical guarantees of our moment matching method. Our analysis uses tools from Jackson's seminal work on approximation with positive polynomial kernels.

Data Structures and Algorithms,Numerical Analysis
Online Spectral Approximation in Random Order Streams

Masataka Gohda,Naonori Kakimura

DOI: https://doi.org/10.48550/arXiv.1911.08800

2019-11-20

Abstract:This paper studies spectral approximation for a positive semidefinite matrix in the online setting. It is known in [Cohen et al. APPROX 2016] that we can construct a spectral approximation of a given $n \times d$ matrix in the online setting if an additive error is allowed. In this paper, we propose an online algorithm that avoids an additive error with the same time and space complexities as the algorithm of Cohen et al., and provides a better upper bound on the approximation size when a given matrix has small rank. In addition, we consider the online random order setting where a row of a given matrix arrives uniformly at random. In this setting, we propose time and space efficient algorithms to find a spectral approximation. Moreover, we reveal that a lower bound on the approximation size in the online random order setting is $\Omega (d \epsilon^{-2} \log n)$, which is larger than the one in the offline setting by an $\mathrm{O}\left( \log n \right)$ factor.

Numerical Analysis,Data Structures and Algorithms
A Fast Spectral Relaxation Approach to Matrix Completion Via Kronecker Products.

Hui Zhao,Jiuqiang Han,Naiyan Wang,Congfu Xu,Zhihua Zhang

DOI: https://doi.org/10.1609/aaai.v25i1.7913

2011-01-01

Abstract:In the existing methods for solving matrix completion, such as singular value thresholding (SVT), soft-impute and ﬁxed point continuation (FPCA) algorithms, it is typically required to repeatedly implement singular value decompositions (SVD) of matrices.When the size of the matrix in question is large, the computational complexity of ﬁnding a solution is costly. To reduce this expensive computational complexity, we apply Kronecker products to handle the matrix completion problem. In particular, we propose using Kronecker factorization, which approximates a matrix by the Kronecker product of several matrices of smaller sizes. Weintroduce Kronecker factorization into the soft-impute framework and devise an effective matrix completion algorithm.Especially when the factorized matrices have about the samesizes, the computational complexity of our algorithm is improved substantially.

Fast Spectral Low Rank Matrix Approximation

Fast randomized algorithms for low-rank matrix approximations with applications in global comparative analysis of a class of data sets

An algorithm for low-rank matrix factorization and its applications

Fast and Accurate Matrix Completion Via Truncated Nuclear Norm Regularization

Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares

Fast Spectrum Estimation of Some Kernel Matrices

Low-Rank Approximations with Sparse Factors I: Basic Algorithms and Error Analysis

Fast Low-Rank Matrix Learning with Nonconvex Regularization

Fast $(1+\varepsilon)$-Approximation Algorithms for Binary Matrix Factorization

Efficient Randomized Algorithms for the Fixed-Precision Low-Rank Matrix Approximation

An Efficient Gauss--Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations

Fast and Low Memory Cost Matrix Factorization: Algorithm, Analysis, and Case Study.

Fast Machine-Precision Spectral Likelihoods for Stationary Time Series

Fast Generalized Matrix Regression with Applications in Machine Learning

A Fast Spectral Algorithm for Mean Estimation with Sub-Gaussian Rates

Accurate Low Rank Approximation at a Low Computational Cost

A generalized Nystrom method with column sketching for low-rank approximation of nonsymmetric matrices

Efficient direct position determination using Frobenius norm approximation

Sublinear Time Spectral Density Estimation

Online Spectral Approximation in Random Order Streams

A Fast Spectral Relaxation Approach to Matrix Completion Via Kronecker Products.