Fast Algorithms for Displacement and Low-Rank Structured Matrices

Shivkumar Chandrasekaran,Nithin Govindarajan,Abhejit Rajagopal

DOI: https://doi.org/10.1145/3208976.3209025

2018-07-10

Abstract:This tutorial provides an introduction to the development of fast matrix algorithms based on the notions of displacement and various low-rank structures.

Numerical Analysis

What problem does this paper attempt to address?

Inversion of Displacement Operators

VY Pan,XM Wang

DOI: https://doi.org/10.1137/s089547980238627x

IF: 1.908

2003-01-01

SIAM Journal on Matrix Analysis and Applications

Abstract:We recall briefly the displacement rank approach to the computations with structured matrices, which we trace back to the seminal paper by Kailath, Kung, and Morf [ J. Math. Anal. Appl., 68 (1979), pp. 395--407]. The concluding stage of the computations is the recovery of the output from its compressed representation via the associated displacement operator L. The recovery amounts to the inversion of the operator. That is, one must express a structured matrix M via its image L(M). We show a general method for obtaining such expressions that works for all displacement operators (under only the mildest nonsingularity assumptions) and thus provides the foundation for the displacement rank approach to practical computations with structured matrices. We also apply our techniques to specify the expressions for various important classes of matrices. Besides unified derivation of several known formulae, we obtain some new ones, in particular, for the matrices associated with the tangential Nevanlinna--Pick problems. This enables acceleration of the known solution algorithms. We show several new matrix representations of the problem in the important confluent case. Finally, we substantially improve the known estimates for the norms of the inverse displacement operators, which are critical numerical parameters for computations based on the displacement approach.
Fast Algorithms for Recovering a Corrupted Low-Rank Matrix

Arvind Ganesh,Zhouchen Lin,John Wright,Leqin Wu,Minming Chen,Yi Ma

DOI: https://doi.org/10.1109/camsap.2009.5413299

2009-01-01

Abstract:This paper studies algorithms for solving the problem of recovering a low-rank matrix with a fraction of its entries arbitrarily corrupted. This problem can be viewed as a robust version of classical PCA, and arises in a number of application domains, including image processing, web data ranking, and bioinformatic data analysis. It was recently shown that under surprisingly broad conditions, it can be exactly solved via a convex programming surrogate that combines nuclear norm minimization and l(1)-norm minimization. This paper develops and compares two complementary approaches for solving this convex program. The first is an accelerated proximal gradient algorithm directly applied to the primal; while the second is a gradient algorithm applied to the dual problem. Both are several orders of magnitude faster than the previous state-of-the-art algorithm for this problem, which was based on iterative thresholding. Simulations demonstrate the performance improvement that can be obtained via these two algorithms, and clarify their relative merits.
Fast Decentralized Federated Low Rank Matrix Recovery from Column-wise Linear Projections

Shana Moothedath,Namrata Vaswani

2024-02-12

Abstract:This work develops a provably accurate fully-decentralized alternating projected gradient descent (GD) algorithm for recovering a low rank (LR) matrix from mutually independent projections of each of its columns, in a fast and communication-efficient fashion. To our best knowledge, this work is the first attempt to develop a provably correct decentralized algorithm (i) for any problem involving the use of an alternating projected GD algorithm; (ii) and for any problem in which the constraint set to be projected to is a non-convex set.

Information Theory
Fast and Low Memory Cost Matrix Factorization: Algorithm, Analysis, and Case Study.

Yan,Mingkui Tan,Ivor W. Tsang,Yi Yang,Qinfeng Shi,Chengqi Zhang

DOI: https://doi.org/10.1109/tkde.2018.2882197

IF: 9.235

2018-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Matrix factorization has been widely applied to various applications. With the fast development of storage and internet technologies, we have been witnessing a rapid increase of data. In this paper, we propose new algorithms for matrix factorization with the emphasis on efficiency. In addition, most existing methods of matrix factorization only consider a general smooth least square loss. Differently, many real-world applications have distinctive characteristics. As a result, different losses should be used accordingly. Therefore, it is beneficial to design new matrix factorization algorithms that are able to deal with both smooth and non-smooth losses. To this end, one needs to analyze the characteristics of target data and use the most appropriate loss based on the analysis. We particularly study two representative cases of low-rank matrix recovery, i.e., collaborative filtering for recommendation and high dynamic range imaging. To solve these two problems, we respectively propose a stage-wise matrix factorization algorithm by exploiting manifold optimization techniques. From our theoretical analysis, they are both are provably guaranteed to converge to a stationary point. Extensive experiments on recommender systems and high dynamic range imaging demonstrate the satisfactory performance and efficiency of our proposed method on large-scale real data.
Learning-Based Low-Rank Approximations

Piotr Indyk,Ali Vakilian,Yang Yuan

2019-01-01

Abstract:We introduce a "learning-based" algorithm for the low-rank decomposition problem: given an n x d matrix A, and a parameter k, compute a rank-k matrix A' that minimizes the approximation loss parallel to A - A'parallel to(F). The algorithm uses a training set of input matrices in order to optimize its performance. Specifically, some of the most efficient approximate algorithms for computing low-rank approximations proceed by computing a projection SA, where S is a sparse random m x n "sketching matrix", and then performing the singular value decomposition of SA. We show how to replace the random matrix S with a "learned" matrix of the same sparsity to reduce the error.Our experiments show that, for multiple types of data sets, a learned sketch matrix can substantially reduce the approximation loss compared to a random matrix S, sometimes by one order of magnitude. We also study mixed matrices where only some of the rows are trained and the remaining ones are random, and show that matrices still offer improved performance while retaining worst-case guarantees.Finally, to understand the theoretical aspects of our approach, we study the special case of m = 1. In particular, we give an approximation algorithm for minimizing the empirical loss, with approximation factor depending on the stable rank of matrices in the training set. We also show generalization bounds for the sketch matrix learning problem.
Scalable Algorithms for Locally Low-Rank Matrix Modeling

Qilong Gu,Joshua D. Trzasko,Arindam Banerjee

DOI: https://doi.org/10.1007/s10115-018-1320-9

IF: 2.7

2019-01-01

Knowledge and Information Systems

Abstract:We consider the problem of modeling data matrices with locally low-rank (LLR) structure, a generalization of the popular low-rank structure widely used in a variety of real-world application domains ranging from medical imaging to recommendation systems. While LLR modeling has been found to be promising in real-world application domains, limited progress has been made on the design of scalable algorithms for such structures. In this paper, we consider a convex relaxation of LLR structure and propose an efficient algorithm based on dual projected gradient descent (D-PGD) for computing the proximal operator. While the original problem is non-smooth, so that primal (sub)gradient algorithms will be slow, we show that the proposed D-PGD algorithm has geometrical convergence rate. We present several practical ways to further speed up the computations, including acceleration and approximate SVD computations. With experiments on both synthetic and real data from MRI (magnetic resonance imaging) denoising, we illustrate the superior performance of the proposed D-PGD algorithm compared to several baselines.
Fast Stochastic Algorithms for Low-rank and Nonsmooth Matrix Problems

Dan Garber,Atara Kaplan

DOI: https://doi.org/10.48550/arXiv.1809.10477

IF: 5.414

2018-09-27

Machine Learning

Abstract:Composite convex optimization problems which include both a nonsmooth term and a low-rank promoting term have important applications in machine learning and signal processing, such as when one wishes to recover an unknown matrix that is simultaneously low-rank and sparse. However, such problems are highly challenging to solve in large-scale: the low-rank promoting term prohibits efficient implementations of proximal methods for composite optimization and even simple subgradient methods. On the other hand, methods which are tailored for low-rank optimization, such as conditional gradient-type methods, which are often applied to a smooth approximation of the nonsmooth objective, are slow since their runtime scales with both the large Lipshitz parameter of the smoothed gradient vector and with $1/\epsilon$. In this paper we develop efficient algorithms for \textit{stochastic} optimization of a strongly-convex objective which includes both a nonsmooth term and a low-rank promoting term. In particular, to the best of our knowledge, we present the first algorithm that enjoys all following critical properties for large-scale problems: i) (nearly) optimal sample complexity, ii) each iteration requires only a single \textit{low-rank} SVD computation, and iii) overall number of thin-SVD computations scales only with $\log{1/\epsilon}$ (as opposed to $\textrm{poly}(1/\epsilon)$ in previous methods). We also give an algorithm for the closely-related finite-sum setting. At the heart of our results lie a novel combination of a variance-reduction technique and the use of a \textit{weak-proximal oracle} which is key to obtaining all above three properties simultaneously.
Fast Algorithms for Recovering a Corrupted

Low-Rank Matrix,Arvind Ganesh,Zhouchen Lin,John Wright,Leqin Wu,Minming Chen,Yi Ma

2009-01-01

Abstract:This paper studies algorithms for solving the prob- lem of recovering a low-rank matrix with a fraction of its entries arbitrarily corrupted. This problem can be viewed as a robust version of classical PCA, and arises in a number of application domains, including image processing, web data ranking, and bioinformatic data analysis. It was recently shown that under surprisingly broad conditions, it can be exactly solved via a convex programming surrogate that combines nuclear norm minimization and ' 1 -norm minimization. This paper develops and compares two complementary approaches for solving this convex program. The first is an accelerated proximal gradient algorithm directly applied to the primal; while the second is a gradient algorithm applied to the dual problem. Both are several orders of magnitude faster than the previous state- of-the-art algorithm for this problem, which was based on iterative thresholding. Simulations demonstrate the performance improvement that can be obtained via these two algorithms, and clarify their relative merits.
Randomized Compression of Rank-Structured Matrices Accelerated with Graph Coloring

James Levitt,Per-Gunnar Martinsson

DOI: https://doi.org/10.1016/j.cam.2024.116044

2024-06-22

Abstract:A randomized algorithm for computing a data sparse representation of a given rank structured matrix $A$ (a.k.a. an $H$-matrix) is presented. The algorithm draws on the randomized singular value decomposition (RSVD), and operates under the assumption that algorithms for rapidly applying $A$ and $A^{*}$ to vectors are available. The algorithm analyzes the hierarchical tree that defines the rank structure using graph coloring algorithms to generate a set of random test vectors. The matrix is then applied to the test vectors, and in a final step the matrix itself is reconstructed by the observed input-output pairs. The method presented is an evolution of the "peeling algorithm" of L. Lin, J. Lu, and L. Ying, "Fast construction of hierarchical matrix representation from matrix-vector multiplication," JCP, 230(10), 2011. For the case of uniform trees, the new method substantially reduces the pre-factor of the original peeling algorithm. More significantly, the new technique leads to dramatic acceleration for many non-uniform trees since it constructs sample vectors that are optimized for a given tree. The algorithm is particularly effective for kernel matrices involving a set of points restricted to a lower dimensional object than the ambient space, such as a boundary integral equation defined on a surface in three dimensions.

Numerical Analysis
Fast construction of hierarchical matrix representation from matrix-vector multiplication

Lin Lin,Jianfeng Lu,Lexing Ying

DOI: https://doi.org/10.48550/arXiv.1001.0149

2010-08-21

Abstract:We develop a hierarchical matrix construction algorithm using matrix-vector multiplications, based on the randomized singular value decomposition of low-rank matrices. The algorithm uses $\mathcal{O}(\log n)$ applications of the matrix on structured random test vectors and $\mathcal{O}(n \log n)$ extra computational cost, where $n$ is the dimension of the unknown matrix. Numerical examples on constructing Green's functions for elliptic operators in two dimensions show efficiency and accuracy of the proposed algorithm.

Numerical Analysis
Fast Gradient Method for Low-Rank Matrix Estimation

Hongyi Li,Zhen Peng,Chengwei Pan,Di Zhao

DOI: https://doi.org/10.1007/s10915-023-02266-7

2023-06-18

Journal of Scientific Computing

Abstract:Projected gradient descent and its Riemannian variant belong to a typical class of methods for low-rank matrix estimation. This paper proposes a new Nesterov's Accelerated Riemannian Gradient algorithm using efficient orthographic retraction and tangent space projection. The subspace relationship between iterative and extrapolated sequences on the low-rank matrix manifold provides computational convenience. With perturbation analysis of truncated singular value decomposition and two retractions, we systematically analyze the local convergence of gradient algorithms and Nesterov's variants in the Euclidean and Riemannian settings. Theoretically, we estimate the exact rate of local linear convergence under different parameters using the spectral radius in a closed form and give the optimal convergence rate and the corresponding momentum parameter. When the parameter is unknown, the adaptive restart scheme can avoid the oscillation problem caused by high momentum, thus approaching the optimal convergence rate. Extensive numerical experiments confirm the estimations of convergence rate and demonstrate that the proposed algorithm is competitive with first-order methods for matrix completion and matrix sensing.

mathematics, applied
Fast Convex Optimization Algorithms for the Recovery of a Corrupted Low-Rank Matrix

arvind ganesh,zhouchen lin,john c wright,leqin wu,yi ma minming chen

2009-01-01

Abstract:This paper studies algorithms for solving the problem of recovering a low-rank matrix with a fraction of its entries arbitrarily corrupted. This problem can be viewed as a robust version of classical PCA, and arises in a number of application domains, including image processing, web data ranking, and bioinformatic data analysis. It was recently shown that under surprisingly broad conditions, it can be exactly solved via a convex programming surrogate that combines nuclear norm minimization and '1-norm minimization. This paper develops and compares two complementary approaches for solving this convex program. The rst is an accelerated proximal gradient algorithm directly applied to the primal; while the second is a gradient algorithm applied to the dual prob- lem. Both are several orders of magnitude faster than the previous state-of-the-art algorithm for this problem, which was based on iterative thresholding. Simulations demonstrate the performance improvement that can be obtained via these two algorithms, and clarify their relative merits.
Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees

Yudong Chen,Martin J. Wainwright

DOI: https://doi.org/10.48550/arXiv.1509.03025

2015-09-10

Abstract:Optimization problems with rank constraints arise in many applications, including matrix regression, structured PCA, matrix completion and matrix decomposition problems. An attractive heuristic for solving such problems is to factorize the low-rank matrix, and to run projected gradient descent on the nonconvex factorized optimization problem. The goal of this problem is to provide a general theoretical framework for understanding when such methods work well, and to characterize the nature of the resulting fixed point. We provide a simple set of conditions under which projected gradient descent, when given a suitable initialization, converges geometrically to a statistically useful solution. Our results are applicable even when the initial solution is outside any region of local convexity, and even when the problem is globally concave. Working in a non-asymptotic framework, we show that our conditions are satisfied for a wide range of concrete models, including matrix regression, structured PCA, matrix completion with real and quantized observations, matrix decomposition, and graph clustering problems. Simulation results show excellent agreement with the theoretical predictions.

Statistics Theory,Machine Learning
A Quadratically Convergent Algorithm for Structured Low-Rank Approximation

Éric Schost,Pierre-Jean Spaenlehauer

DOI: https://doi.org/10.1007/s10208-015-9256-x

IF: 3.4388

2015-03-11

Foundations of Computational Mathematics

Abstract:Structured Low-Rank Approximation is a problem arising in a wide range of applications in Numerical Analysis and Engineering Sciences. Given an input matrix M$$M$$, the goal is to compute a matrix M′$$M'$$ of given rank r$$r$$ in a linear or affine subspace E$$E$$ of matrices (usually encoding a specific structure) such that the Frobenius distance M-M′$$\left\| M-M' \right\| $$ is small. We propose a Newton-like iteration for solving this problem, whose main feature is that it converges locally quadratically to such a matrix under mild transversality assumptions between the manifold of matrices of rank r$$r$$ and the linear/affine subspace E$$E$$. We also show that the distance between the limit of the iteration and the optimal solution of the problem is quadratic in the distance between the input matrix and the manifold of rank r$$r$$ matrices in E$$E$$. To illustrate the applicability of this algorithm, we propose a Maple implementation and give experimental results for several applicative problems that can be modeled by Structured Low-Rank Approximation: univariate approximate GCDs (Sylvester matrices), low-rank matrix completion (coordinate spaces) and denoising procedures (Hankel matrices).

mathematics, applied,computer science, theory & methods
An algorithm for low-rank matrix factorization and its applications

Baiyu Chen,Zi Yang,Zhouwang Yang

DOI: https://doi.org/10.1016/j.neucom.2017.09.052

IF: 6

2018-01-01

Neurocomputing

Abstract:This paper proposes a valid and fast algorithm for low-rank matrix factorization. There are multiple applications for low-rank matrix factorization, and numerous algorithms have been developed to solve this problem. However, many algorithms do not use rank directly; instead, they minimize a nuclear norm by using Singular Value Decomposition (SVD), which requires a huge time cost. In addition, these algorithms often fix the dimension of the factorized matrix, meaning that one must first find an optimum dimension for the factorized matrix in order to obtain a solution. Unfortunately, the optimum dimension is unknown in many practical problems, such as matrix completion and recommender systems. Therefore, it is necessary to develop a faster algorithm that can also estimate the optimum dimension. In this paper, we use the Hidden Matrix Factorized Augmented Lagrangian Method to solve low-rank matrix factorizations. We also add a tool to dynamically estimate the optimum dimension and adjust it while simultaneously running the algorithm. Additionally, in the era of Big Data, there will be more and more large, sparse data. In face of such highly sparse data, our algorithm has the potential to be more effective than other algorithms. We applied it to some practical problems, e.g. Low-Rank Representation(LRR), and matrix completion with constraint. In numerical experiments, it has performed well when applied to both synthetic data and real-world data.

computer science, artificial intelligence
Finite Projective Geometry based Fast, Conflict-free Parallel Matrix Computations

Shreeniwas Sapre,Hrishikesh Sharma,Abhishek Patil,B. S. Adiga,Sachin Patkar

DOI: https://doi.org/10.48550/arXiv.1107.1127

2011-07-05

Numerical Analysis

Abstract:Matrix computations, especially iterative PDE solving (and the sparse matrix vector multiplication subproblem within) using conjugate gradient algorithm, and LU/Cholesky decomposition for solving system of linear equations, form the kernel of many applications, such as circuit simulators, computational fluid dynamics or structural analysis etc. The problem of designing approaches for parallelizing these computations, to get good speedups as much as possible as per Amdahl's law, has been continuously researched upon. In this paper, we discuss approaches based on the use of finite projective geometry graphs for these two problems. For the problem of conjugate gradient algorithm, the approach looks at an alternative data distribution based on projective-geometry concepts. It is proved that this data distribution is an optimal data distribution for scheduling the main problem of dense matrix-vector multiplication. For the problem of parallel LU/Cholesky decomposition of general matrices, the approach is motivated by the recently published scheme for interconnects of distributed systems, perfect difference networks. We find that projective-geometry based graphs indeed offer an exciting way of parallelizing these computations, and in fact many others. Moreover, their applications ranges from architectural ones (interconnect choice) to algorithmic ones (data distributions).
Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview

Yuejie Chi,Yue M. Lu,Yuxin Chen

DOI: https://doi.org/10.1109/TSP.2019.2937282

IF: 4.875

2022-03-15

IEEE Transactions on Signal Processing

Abstract:Substantial progress has been made recently on developing provably accurate and efficient algorithms for low-rank matrix factorization via nonconvex optimization. While conventional wisdom often takes a dim view of nonconvex optimization algorithms due to their susceptibility to spurious local minima, simple iterative methods such as gradient descent have been remarkably successful in practice. The theoretical footings, however, had been largely lacking until recently. In this tutorial-style overview, we highlight the important role of statistical models in enabling efficient nonconvex optimization with performance guarantees. We review two contrasting approaches: (1) two-stage algorithms, which consist of a tailored initialization step followed by successive refinement; and (2) global landscape analysis and initialization-free algorithms. Several canonical matrix factorization problems are discussed, including but not limited to matrix sensing, phase retrieval, matrix completion, blind deconvolution, and robust principal component analysis. Special care is taken to illustrate the key technical insights underlying their analyses. This article serves as a testament that the integrated consideration of optimization and statistics leads to fruitful research findings.

engineering, electrical & electronic
Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares

Trevor Hastie,Rahul Mazumder,Jason Lee,Reza Zadeh

DOI: https://doi.org/10.48550/arXiv.1410.2596

2014-10-09

Methodology

Abstract:The matrix-completion problem has attracted a lot of attention, largely as a result of the celebrated Netflix competition. Two popular approaches for solving the problem are nuclear-norm-regularized matrix approximation (Candes and Tao, 2009, Mazumder, Hastie and Tibshirani, 2010), and maximum-margin matrix factorization (Srebro, Rennie and Jaakkola, 2005). These two procedures are in some cases solving equivalent problems, but with quite different algorithms. In this article we bring the two approaches together, leading to an efficient algorithm for large matrix factorization and completion that outperforms both of these. We develop a software package "softImpute" in R for implementing our approaches, and a distributed version for very large matrices using the "Spark" cluster programming environment.
FMMTL: FMM Template Library A Generalized Framework for Kernel Matrices

Cris Cecka,Simon Layton

DOI: https://doi.org/10.1007/978-3-319-10705-9_60

2014-10-31

Abstract:In response to two decades of development in structured dense matrix algorithms and a vast number of research codes, we present designs and progress towards a codebase that is abstracted over the primary domains of research. In the domain of mathematics, this includes the development of interaction kernels and their low-rank expansions. In the domain of high performance computing, this includes the optimized construction, traversal, and scheduling algorithms for the appropriate operations. We present a versatile system that can encompass the design decisions made over a decade of research while providing an abstracted, intuitive, and usable front-end that can integrated into existing linear algebra libraries.
Fast Multiresolution Algorithms for Solving Linear Equations: A Comparative Study

Francesc Arandiga,Vicente F. Candela,Rosa Donat

DOI: https://doi.org/10.1137/0916037

IF: 2.968

1995-05-01

SIAM Journal on Scientific Computing

Abstract:In [G. Beylkin, R. Coifman, and V. Rokhlin, Comm. Pure Appl. Math., XLIV (1991), PP. 141183] and [B. Engquist, S. Osher, and S. Zhong, SIAM J. Sci. Comput., 15 (1994), pp. 755775], orthonormal wavelet basis and the multiscale analysis they define are used as building blocks in the design of algorithms for the rapid numerical application of a number of linear operators to arbitrary vectors. The algorithms may be viewed as a method for converting (whenever possible) dense matrices to sparse form. We use the discrete multiresolution analysis described by Harten in [J. Appl. Numer Math., 12 (1993), pp. 153193] as the building block in the algorithms and compare their performance with the wavelet-based ones. Computationally, both techniques give comparable results.

mathematics, applied

Fast Algorithms for Displacement and Low-Rank Structured Matrices

Inversion of Displacement Operators

Fast Algorithms for Recovering a Corrupted Low-Rank Matrix

Fast Decentralized Federated Low Rank Matrix Recovery from Column-wise Linear Projections

Fast and Low Memory Cost Matrix Factorization: Algorithm, Analysis, and Case Study.

Learning-Based Low-Rank Approximations

Scalable Algorithms for Locally Low-Rank Matrix Modeling

Fast Stochastic Algorithms for Low-rank and Nonsmooth Matrix Problems

Fast Algorithms for Recovering a Corrupted

Randomized Compression of Rank-Structured Matrices Accelerated with Graph Coloring

Fast construction of hierarchical matrix representation from matrix-vector multiplication

Fast Gradient Method for Low-Rank Matrix Estimation

Fast Convex Optimization Algorithms for the Recovery of a Corrupted Low-Rank Matrix

Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees

A Quadratically Convergent Algorithm for Structured Low-Rank Approximation

An algorithm for low-rank matrix factorization and its applications

Finite Projective Geometry based Fast, Conflict-free Parallel Matrix Computations

Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview

Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares

FMMTL: FMM Template Library A Generalized Framework for Kernel Matrices

Fast Multiresolution Algorithms for Solving Linear Equations: A Comparative Study