Some Software Packages for Partial SVD Computation

Zhouchen Lin

2011-01-01

Abstract:This technical report introduces some software packages for partial SVD computation, including optimized PROPACK, modified PROPACK for computing singular values above a threshold and the corresponding singular vectors, and block Lanczos with warm start (BLWS). The current version is preliminary. The details will be enriched soon.

What problem does this paper attempt to address?

PRIMME_SVDS: A High-Performance Preconditioned SVD Solver for Accurate Large-Scale Computations

Lingfei Wu,Eloy Romero,Andreas Stathopoulos

DOI: https://doi.org/10.48550/arXiv.1607.01404

2017-01-25

Abstract:The increasing number of applications requiring the solution of large scale singular value problems have rekindled interest in iterative methods for the SVD. Some promising recent ad- vances in large scale iterative methods are still plagued by slow convergence and accuracy limitations for computing smallest singular triplets. Furthermore, their current implementations in MATLAB cannot address the required large problems. Recently, we presented a preconditioned, two-stage method to effectively and accurately compute a small number of extreme singular triplets. In this research, we present a high-performance software, PRIMME SVDS, that implements our hybrid method based on the state-of-the-art eigensolver package PRIMME for both largest and smallest singular values. PRIMME SVDS fills a gap in production level software for computing the partial SVD, especially with preconditioning. The numerical experiments demonstrate its superior performance compared to other state-of-the-art software and its good parallel performance under strong and weak scaling.

Mathematical Software,Distributed, Parallel, and Cluster Computing,Numerical Analysis
A Novel Fully Hardware-Implemented SVD Solver Based on Ultra-Parallel BCV Jacobi Algorithm

Tang Hu,Xiangdi Li,Xiao Yu,Songnan Ren,Li Yan,Xuyang Bai,Zhiwei Xu,Shiqiang Zhu

DOI: https://doi.org/10.1109/tcsii.2022.3200750

2022-01-01

IEEE Transactions on Circuits & Systems II Express Briefs

Abstract:Efficient FPGA-based floating-point singular value decomposition (SVD) is challenging for its enormous complexity with the rapid growth of the matrix dimension. Numerous hardware architectures have been proposed to improve the performance of SVD by increasing capacity of computation units, reusing data, and enhancing bandwidth. These designs, however, are not optimum due to their low parallelism, poor data access efficiency, and inferior iterations scheduling. In this express, we propose a block column vector Hestenes-Jacobi (BCV Jacobi) algorithm that decomposes an arbitrary large matrix into several blocks, enhances the access efficiency by customizing the distinctive data structure, and improves the system-level parallelism by simplifying the iteration scheduling. The proposed BCV Jacobi algorithm also achieves better scalability and efficiency. Experimental results show that the performance of the proposed FPGA based SVD processor is superior to other SVD implementations in terms of parallelism, data access efficiency, matrix size, and execution time. When compared with state of the art SVD accelerator engine, the proposed algorithm speeds up the runtime over $2{\times }$ on average.
Limited Memory Block Krylov Subspace Optimization for Computing Dominant Singular Value Decompositions.

Xin Liu,Zaiwen Wen,Yin Zhang

DOI: https://doi.org/10.1137/120871328

IF: 2.968

2013-01-01

SIAM Journal on Scientific Computing

Abstract:In many data-intensive applications, the use of principal component analysis and other related techniques is ubiquitous for dimension reduction, data mining, or other transformational purposes. Such transformations often require efficiently, reliably, and accurately computing dominant singular value decompositions (SVDs) of large and dense matrices. In this paper, we propose and study a subspace optimization technique for significantly accelerating the classic simultaneous iteration method. We analyze the convergence of the proposed algorithm and numerically compare it with several state-of-the-art SVD solvers under the MATLAB environment. Extensive computational results show that on a wide range of large unstructured dense matrices, the proposed algorithm can often provide improved efficiency or robustness over existing algorithms.
Accelerating large partial EVD/SVD calculations by filtered block Davidson methods

Yunkai Zhou,Zheng Wang,Aihui Zhou

DOI: https://doi.org/10.1007/s11425-016-0274-0

2016-06-13

Science China Mathematics

Abstract:Partial eigenvalue decomposition (PEVD) and partial singular value decomposition (PSVD) of large sparse matrices are of fundamental importance in a wide range of applications, including latent semantic indexing, spectral clustering, and kernel methods for machine learning. The more challenging problems are when a large number of eigenpairs or singular triplets need to be computed. We develop practical and efficient algorithms for these challenging problems. Our algorithms are based on a filter-accelerated block Davidson method. Two types of filters are utilized, one is Chebyshev polynomial filtering, the other is rational-function filtering by solving linear equations. The former utilizes the fastest growth of the Chebyshev polynomial among same degree polynomials; the latter employs the traditional idea of shift-invert, for which we address the important issue of automatic choice of shifts and propose a practical method for solving the shifted linear equations inside the block Davidson method. Our two filters can efficiently generate high-quality basis vectors to augment the projection subspace at each Davidson iteration step, which allows a restart scheme using an active projection subspace of small dimension. This makes our algorithms memory-economical, thus practical for large PEVD/PSVD calculations. We compare our algorithms with representative methods, including ARPACK, PROPACK, the randomized SVD method, and the limited memory SVD method. Extensive numerical tests on representative datasets demonstrate that, in general, our methods have similar or faster convergence speed in terms of CPU time, while requiring much lower memory comparing with other methods. The much lower memory requirement makes our methods more practical for large-scale PEVD/PSVD computations.

mathematics, applied
svds-C: A Multi-Thread C Code for Computing Truncated Singular Value Decomposition

Xu Feng,Wenjian Yu,Yuyang Xie

2024-05-29

Abstract:This article presents svds-C, an open-source and high-performance C program for accurately and robustly computing truncated SVD, e.g. computing several largest singular values and corresponding singular vectors. We have re-implemented the algorithm of svds in Matlab in C based on MKL or OpenBLAS and multi-thread computing to obtain the parallel program named svds-C. svds-C running on shared-memory computer consumes less time and memory than svds thanks to careful implementation of multi-thread parallelization and memory management. Numerical experiments on different test cases which are synthetically generated or directly from real world datasets show that, svds-C runs remarkably faster than svds with averagely 4.7X and at most 12X speedup for 16-thread parallel computing on a computer with Intel CPU, while preserving same accuracy and consuming about half memory space. Experimental results also demonstrate that svds-C has similar advantages over svds on the computer with AMD CPU, and outperforms other state-of-the-art algorithms for truncated SVD on computing time and robustness.

Mathematical Software
A Hybrid Algorithm for Computing a Partial Singular Value Decomposition Satisfying a Given Threshold

James Baglama,Jonathan A. Chávez Casillas,Vasilije Perović

2024-08-03

Abstract:In this paper, we describe a new hybrid algorithm for computing all singular triplets above a given threshold and provide its implementation in MATLAB/Octave and R. The high performance of our codes and ease at which they can be used, either independently or within a larger numerical scheme, are illustrated through several numerical examples with applications to matrix completion and image compression. Well-documented MATLAB and R codes are provided for public use.

Numerical Analysis
A mixed precision Jacobi SVD algorithm

Weiguo Gao,Yuxin Ma,Meiyue Shao

2024-11-12

Abstract:We propose a mixed precision Jacobi algorithm for computing the singular value decomposition (SVD) of a dense matrix. After appropriate preconditioning, the proposed algorithm computes the SVD in a lower precision as an initial guess, and then performs one-sided Jacobi rotations in the working precision as iterative refinement. By carefully transforming a lower precision solution to a higher precision one, our algorithm achieves about 2 times speedup on the x86-64 architecture compared to the usual one-sided Jacobi SVD algorithm in LAPACK, without sacrificing the accuracy.

Numerical Analysis
Towards a more robust algorithm for computing the restricted singular value decomposition

Ian N. Zwaan

DOI: https://doi.org/10.48550/arXiv.2002.04828

2020-02-12

Abstract:A new algorithm to compute the restricted singular value decomposition of dense matrices is presented. Like Zha's method \cite{Zha92}, the new algorithm uses an implicit Kogbetliantz iteration, but with four major innovations. The first innovation is a useful quasi-upper triangular generalized Schur form that just requires orthonormal transformations to compute. Depending on the application, this Schur form can be used instead of the full decomposition. The second innovation is a new preprocessing phase that requires fewer rank determinations than previous methods. The third innovation is a numerically stable RSVD algorithm for $2\times 2$ upper-triangular matrices, which forms a key component of the implicit Kogbetliantz iteration. The fourth innovation is an alternative scaling for the restricted singular triplets that results in elegant formulas for their computation. Beyond these four innovations, the qualitative (numerical) characteristics of the algorithm are discussed extensively. Some numerical challenges in the (optional) postprocessing phase are considered too; though, their solutions require further research. Numerical tests and examples confirm the effectiveness of the method.

Numerical Analysis
Randomized Algorithms for Solving Singular Value Decomposition Problems with Matlab Toolbox

Xiaowen Li

2024-02-27

Abstract:This thesis gives an overview of the state-of-the-art randomized linear algebra algorithms for singular value decomposition (SVD), including the presentation of existing pseudo-codes and theoretical error analysis. Our main focus is on presenting numerical experiments illustrating image restoration using various randomized singular value decomposition (RSVD) methods; theoretical error bounds, computed errors, and canonical angles analysis for these RSVD algorithms. This thesis also comes with a newly developed Matlab toolbox that contains implementations and test examples for some of the state-of-the-art randomized numerical linear algebra algorithms.

Optimization and Control
Fast ADMM for Sum-of-Squares Programs Using Partial Orthogonality.

Yang Zheng,Giovanni Fantuzzi,Antonis Papachristodoulou

DOI: https://doi.org/10.1109/tac.2018.2886170

IF: 6.549

2018-01-01

IEEE Transactions on Automatic Control

Abstract:When sum-of-squares (SOS) programs are recast as semidefinite programs (SDPs) using the standard monomial basis, the constraint matrices in the SDP possess a structural property that we call partial orthogonality. In this paper, we leverage partial orthogonality to develop a fast first-order method, based on the alternating direction method of multipliers (ADMM), for the solution of the homogeneous self-dual embedding of SDPs describing SOS programs. Precisely, we show how a “diagonal plus low rank” structure implied by partial orthogonality can be exploited to project efficiently the iterates of a recent ADMM algorithm for generic conic programs onto the set defined by the affine constraints of the SDP. The resulting algorithm, implemented as a new package in the solver CDCS, is tested on a range of large-scale SOS programs arising from constrained polynomial optimization problems and from Lyapunov stability analysis of polynomial dynamical systems. These numerical experiments demonstrate the effectiveness of our approach compared to common state-of-the-art solvers.
Generalizing the SVD of a matrix under non-standard inner product and its applications to linear ill-posed problems

Haibo Li

2023-12-16

Abstract:The singular value decomposition (SVD) of a matrix is a powerful tool for many matrix computation problems. In this paper, we consider generalizing the standard SVD to analyze and compute the regularized solution of linear ill-posed problems that arise from discretizing the first kind Fredholm integral equations. For the commonly used quadrature method for discretization, a regularizer of the form $\|x\|_{M}^2:=x^TMx$ should be exploited, where $M$ is symmetric positive definite. To handle this regularizer, we give the weighted SVD (WSVD) of a matrix under the $M$-inner product. Several important applications of WSVD, such as low-rank approximation and solving the least squares problems with minimum $\|\cdot\|_M$-norm, are studied. We propose the weighted Golub-Kahan bidiagonalization (WGKB) to compute several dominant WSVD components and a corresponding weighted LSQR algorithm to iteratively solve the least squares problem. All the above tools and methods are used to analyze and solve linear ill-posed problems with the regularizer $\|x\|_{M}^2$. A WGKB-based subspace projection regularization method is proposed to efficiently compute a good regularized solution, which can incorporate the prior information about $x$ encoded by the regularizer $\|x\|_{M}^2$. Several numerical experiments are performed to illustrate the fruitfulness of our methods.

Numerical Analysis
Preliminary Application of Software Package Gpps to Spare Linear Systems from Meso Scale Simulation of Concrete Specimen

Jianping Wu,Huaifa Ma,Jun Zhao,Shuchang Wang

DOI: https://doi.org/10.4028/www.scientific.net/amm.580-583.2907

2014-01-01

Applied Mechanics and Materials

Abstract:The general parallel preconditioning subspace iteration software GPPS is developed based on the integration of various preconditioning methods, parallelization techniques, and subspace iterations, to solve general sparse linear systems. The software has several characteristics, including high computational performance, good readability, simple interface, and excellent scalability. In this paper, the functions of the software are outlined, and then it is used to solve the sparse linear systems from meso-scale simulation of concrete specimens. The numerical experiments show that GPPS is clearly superior to the software package AZTEC, which is used in the simulation up to now, and in addition, ICT and the parallelization technique based on factors combination outperform other preconditioning techniques integrated in the software.
High Order Numerical Methods To Approximate The Singular Value Decomposition

Diego Armentano,Jean-Claude Yakoubsohn

2023-09-12

Abstract:In this paper, we present a class of high order methods to approximate the singular value decomposition of a given complex matrix (SVD). To the best of our knowledge, only methods up to order three appear in the the literature. A first part is dedicated to defline and analyse this class of method in the regular case, i.e., when the singular values are pairwise distinct. The construction is based on a perturbation analysis of a suitable system of associated to the SVD (SVD system). More precisely, for an integer $p$ be given, we define a sequence which converges with an order $p + 1$ towards the left-right singular vectors and the singular values if the initial approximation of the SVD system satisfies a condition which depends on three quantities : the norm of initial approximation of the SVD system, the greatest singular value and the greatest inverse of the modulus of the difference between the singular values. From a numerical computational point of view, this furnishes a very efficient simple test to prove and certifiy the existence of a SVD in neighborhood of the initial approximation. We generalize these result in the case of clusters of singular values. We show also how to use the result of regular case to detect the clusters of singular values and to define a notion of deflation of the SVD. Moreover numerical experiments confirm the theoretical results.

Numerical Analysis
A Block Lanczos with Warm Start Technique for Accelerating Nuclear Norm Minimization Algorithms

Zhouchen Lin,Siming Wei

2010-01-01

Abstract:Recent years have witnessed the popularity of using rank minimization as a regularizer for various signal processing and machine learning problems. As rank minimization problems are often converted to nuclear norm minimization (NNM) problems, they have to be solved iteratively and each iteration requires computing a singular value decomposition (SVD). Therefore, their solution suffers from the high computation cost of multiple SVDs. To relieve this issue, we propose using the block Lanczos method to compute the partial SVDs, where the principal singular subspaces obtained in the previous iteration are used to start the block Lanczos procedure. To avoid the expensive reorthogonalization in the Lanczos procedure, the block Lanczos procedure is performed for only a few steps. Our block Lanczos with warm start (BLWS) technique can be adopted by different algorithms that solve NNM problems. We present numerical results on applying BLWS to Robust PCA and Matrix Completion problems. Experimental results show that our BLWS technique usually accelerates its host algorithm by at least two to three times.
The Singular Value Decomposition, Applications and Beyond.

Zhihua Zhang

DOI: https://doi.org/10.48550/arxiv.1510.08532

2015-01-01

Abstract:The singular value decomposition (SVD) is not only a classical theory in matrix computation and analysis, but also is a powerful tool in machine learning and modern data analysis. In this tutorial we first study the basic notion of SVD and then show the central role of SVD in matrices. Using majorization theory, we consider variational principles of singular values and eigenvalues. Built on SVD and a theory of symmetric gauge functions, we discuss unitarily invariant norms, which are then used to formulate general results for matrix low rank approximation. We study the subdifferentials of unitarily invariant norms. These results would be potentially useful in many machine learning problems such as matrix completion and matrix data classification. Finally, we discuss matrix low rank approximation and its recent developments such as randomized SVD, approximate matrix multiplication, CUR decomposition, and Nystrom approximation. Randomized algorithms are important approaches to large scale SVD as well as fast matrix computations.
A Fast deflation Method for Sparse Principal Component Analysis via Subspace Projections

Cong Xu,Min Yang,Jin Zhang

DOI: https://doi.org/10.48550/arXiv.1912.01449

IF: 5.414

2019-12-03

Machine Learning

Abstract:The implementation of conventional sparse principal component analysis (SPCA) on high-dimensional data sets has become a time consuming work. In this paper, a series of subspace projections are constructed efficiently by using Household QR factorization. With the aid of these subspace projections, a fast deflation method, called SPCA-SP, is developed for SPCA. This method keeps a good tradeoff between various criteria, including sparsity, orthogonality, explained variance, balance of sparsity, and computational cost. Comparative experiments on the benchmark data sets confirm the effectiveness of the proposed method.
Using Cross-Product Matrices to Compute the SVD.

Zhongxiao Jia

DOI: https://doi.org/10.1007/s11075-006-9022-x

IF: 2.37

2006-01-01

Numerical Algorithms

Abstract:This paper concerns accurate computation of the singular value decomposition (SVD) of an $m\times n$ matrix $A$. As is well known, cross-product matrix based SVD algorithms compute large singular values accurately but generally deliver poor small singular values. A new novel cross-product matrix based SVD method is proposed: (a) Use a backward stable algorithm to compute the eigenpairs of $A^{\rm T}A$ and take the square roots of the large eigenvalues of it as the large singular values of $A$ ; (b) form the Rayleigh quotient of $A^{\rm T}A$ with respect to the matrix consisting of the computed eigenvectors associated with the computed small eigenvalues of $A^{\rm T}A$ ; (c) compute the eigenvalues of the Rayleigh quotient and take the square roots of them as the small singular values of $A$. A detailed quantitative error analysis is conducted on the method. It is proved that if small singular values are well separated from the large ones then the method can compute the small ones accurately up to the order of the unit roundoff $\epsilon$. An algorithm is developed that is not only cheaper than the standard Golub–Reinsch and Chan SVD algorithms but also can update or downdate a new SVD by adding or deleting a row and compute certain refined Ritz vectors for large matrix eigenproblems at very low cost. Several variants of the algorithm are proposed that compute some or all parts of the SVD. Typical numerical examples confirm the high accuracy of our algorithm.
SDPNAL+: A Matlab software for semidefinite programming with bound constraints (version 1.0)

Defeng Sun,Kim-Chuan Toh,Yancheng Yuan,Xin-Yuan Zhao

DOI: https://doi.org/10.1080/10556788.2019.1576176

2019-02-22

Optimization Methods and Software

Abstract:Sdpnal+ is a MATLAB software package that implements an augmented Lagrangian based method to solve large scale semidefinite programming problems with bound constraints. The implementation was initially based on a majorized semismooth Newton-CG augmented Lagrangian method, here we designed it within an inexact symmetric Gauss-Seidel based semi-proximal ADMM/ALM (alternating direction method of multipliers/augmented Lagrangian method) framework for the purpose of deriving simpler stopping conditions and closing the gap between the practical implementation of the algorithm and the theoretical algorithm. The basic code is written in MATLAB, but some subroutines in C language are incorporated via Mex files. We also design a convenient interface for users to input their SDP models into the solver. Numerous problems arising from combinatorial optimization and binary integer quadratic programming problems have been tested to evaluate the performance of the solver. Extensive numerical experiments conducted in [L.Q. Yang, D.F. Sun, and K.C. Toh, SDPNAL+: A majorized semismooth Newton-CG augmented Lagrangian method for semidefinite programming with nonnegative constraints, Math. Program. Comput. 7 (2015), pp. 331–366] show that the proposed method is quite efficient and robust, in that it is able to solve 98.9% of the 745 test instances of SDP problems arising from various applications to the accuracy of in the relative KKT residual.

operations research & management science,mathematics, applied,computer science, software engineering
An Augmented Matrix-Based CJ-FEAST SVDsolver for Computing a Partial Singular Value Decomposition with the Singular Values in a Given Interval

Zhongxiao Jia,Kailiang Zhang

DOI: https://doi.org/10.1137/23m1547500

IF: 1.908

2024-01-05

SIAM Journal on Matrix Analysis and Applications

Abstract:SIAM Journal on Matrix Analysis and Applications, Volume 45, Issue 1, Page 24-58, March 2024. The cross-product matrix-based CJ-FEAST SVDsolver proposed previously by the authors is shown to compute the left singular vector possibly much less accurately than the right singular vector and may be numerically backward unstable when a desired singular value is small. In this paper, an alternative augmented matrix-based CJ-FEAST SVDsolver is proposed to compute the singular triplets of a large matrix [math] with the singular values in an interval [math] contained in the singular spectrum. The new CJ-FEAST SVDsolver is a subspace iteration applied to an approximate spectral projector of the augmented matrix [math] associated with the eigenvalues in [math], and it constructs approximate left and right singular subspaces independently, onto which [math] is projected to obtain the Ritz approximations to the desired singular triplets. Compact estimates are given for the accuracy of the approximate spectral projector constructed by the Chebyshev–Jackson series expansion in terms of series degree, and a number of convergence results are established. The new solver is proved to be always numerically backward stable. A convergence comparison of the cross-product-based and augmented matrix-based CJ-FEAST SVDsolvers is made, and a general-purpose choice strategy between the two solvers is proposed for the robustness and overall efficiency. Numerical experiments confirm all the results and meanwhile demonstrate that the proposed solver is more robust and substantially more efficient than the corresponding contour integral-based versions that exploit the trapezoidal rule and the Gauss–Legendre quadrature to construct an approximate spectral projector.

mathematics, applied
PyParSVD: A streaming, distributed and randomized singular-value-decomposition library

Romit Maulik,Gianmarco Mengaldo

DOI: https://doi.org/10.48550/arXiv.2108.08845

2021-08-19

Abstract:We introduce PyParSVD\footnote{<a class="link-external link-https" href="https://github.com/Romit-Maulik/PyParSVD" rel="external noopener nofollow">this https URL</a>}, a Python library that implements a streaming, distributed and randomized algorithm for the singular value decomposition. To demonstrate its effectiveness, we extract coherent structures from scientific data. Futhermore, we show weak scaling assessments on up to 256 nodes of the Theta machine at Argonne Leadership Computing Facility, demonstrating potential for large-scale data analyses of practical data sets.

Mathematical Software,Distributed, Parallel, and Cluster Computing,Atmospheric and Oceanic Physics,Fluid Dynamics

Some Software Packages for Partial SVD Computation

PRIMME_SVDS: A High-Performance Preconditioned SVD Solver for Accurate Large-Scale Computations

A Novel Fully Hardware-Implemented SVD Solver Based on Ultra-Parallel BCV Jacobi Algorithm

Limited Memory Block Krylov Subspace Optimization for Computing Dominant Singular Value Decompositions.

Accelerating large partial EVD/SVD calculations by filtered block Davidson methods

svds-C: A Multi-Thread C Code for Computing Truncated Singular Value Decomposition

A Hybrid Algorithm for Computing a Partial Singular Value Decomposition Satisfying a Given Threshold

A mixed precision Jacobi SVD algorithm

Towards a more robust algorithm for computing the restricted singular value decomposition

Randomized Algorithms for Solving Singular Value Decomposition Problems with Matlab Toolbox

Fast ADMM for Sum-of-Squares Programs Using Partial Orthogonality.

Generalizing the SVD of a matrix under non-standard inner product and its applications to linear ill-posed problems

Preliminary Application of Software Package Gpps to Spare Linear Systems from Meso Scale Simulation of Concrete Specimen

High Order Numerical Methods To Approximate The Singular Value Decomposition

A Block Lanczos with Warm Start Technique for Accelerating Nuclear Norm Minimization Algorithms

The Singular Value Decomposition, Applications and Beyond.

A Fast deflation Method for Sparse Principal Component Analysis via Subspace Projections

Using Cross-Product Matrices to Compute the SVD.

SDPNAL+: A Matlab software for semidefinite programming with bound constraints (version 1.0)

An Augmented Matrix-Based CJ-FEAST SVDsolver for Computing a Partial Singular Value Decomposition with the Singular Values in a Given Interval

PyParSVD: A streaming, distributed and randomized singular-value-decomposition library