Abstract:Matrices are exceptionally useful in various fields of study as they provide a convenient framework to organize and manipulate data in a structured manner. However, modern matrices can involve billions of elements, making their storage and processing quite demanding in terms of computational resources and memory usage. Although prohibitively large, such matrices are often approximately low rank. We propose an algorithm that exploits this structure to obtain a low rank decomposition of any matrix $\mathbf{A}$ as $\mathbf{A} \approx \mathbf{L}\mathbf{R}$, where $\mathbf{L}$ and $\mathbf{R}$ are the low rank factors. The total number of elements in $\mathbf{L}$ and $\mathbf{R}$ can be significantly less than that in $\mathbf{A}$. Furthermore, the entries of $\mathbf{L}$ and $\mathbf{R}$ are quantized to low precision formats $--$ compressing $\mathbf{A}$ by giving us a low rank and low precision factorization. Our algorithm first computes an approximate basis of the range space of $\mathbf{A}$ by randomly sketching its columns, followed by a quantization of the vectors constituting this basis. It then computes approximate projections of the columns of $\mathbf{A}$ onto this quantized basis. We derive upper bounds on the approximation error of our algorithm, and analyze the impact of target rank and quantization bit-budget. The tradeoff between compression ratio and approximation accuracy allows for flexibility in choosing these parameters based on specific application requirements. We empirically demonstrate the efficacy of our algorithm in image compression, nearest neighbor classification of image and text embeddings, and compressing the layers of LlaMa-$7$b. Our results illustrate that we can achieve compression ratios as aggressive as one bit per matrix coordinate, all while surpassing or maintaining the performance of traditional compression techniques.

Painless BreakupsEfficient Demixing of Low Rank Matrices

Painless Breakups -- Efficient Demixing of Low Rank Matrices

Sparse and Low-Rank Matrix Decompositions

Matrix Compression via Randomized Low Rank and Low Precision Factorization

Efficient algorithms for Tucker decomposition via approximate matrix multiplication

Fast and Low Memory Cost Matrix Factorization: Algorithm, Analysis, and Case Study.

Online randomized interpolative decomposition with a posteriori error estimator for temporal PDE data reduction

A Fast Rank-Reduction Algorithm for Three-Dimensional Seismic Data Interpolation

Low-Rank Kernel Matrix Approximation Using Skeletonized Interpolation With Endo- or Exo-Vertices

Rank-one matrix estimation: analysis of algorithmic and information theoretic limits by the spatial coupling method

Computing Low-Rank Approximation of a Dense Matrix on Multicore CPUs with a GPU and Its Application to Solving a Hierarchically Semiseparable Linear System of Equations

Guarantees of Riemannian Optimization for Low Rank Matrix Recovery

A Separable Surrogate Function Method for Sparse and Low-Rank Matrices Decomposition

Efficient Algorithms for Positive Semi-Definite Total Least Squares Problems, Minimum Rank Problem and Correlation Matrix Computation

Vandermonde Decomposition of Multilevel Toeplitz Matrices with Application to Multidimensional Super-Resolution.

Leveraging subspace information for low-rank matrix reconstruction

Generalized Nonconvex Nonsmooth Low-Rank Matrix Recovery Framework with Feasible Algorithm Designs and Convergence Analysis.

Robust Sensing of Low-Rank Matrices with Non-Orthogonal Sparse Decomposition

Simultaneously Sparse and Low-Rank Abundance Matrix Estimation for Hyperspectral Image Unmixing

Non-convex matrix sensing: Breaking the quadratic rank barrier in the sample complexity

Deinsum: Practically I/O Optimal Multilinear Algebra