Abstract:Matrices are exceptionally useful in various fields of study as they provide a convenient framework to organize and manipulate data in a structured manner. However, modern matrices can involve billions of elements, making their storage and processing quite demanding in terms of computational resources and memory usage. Although prohibitively large, such matrices are often approximately low rank. We propose an algorithm that exploits this structure to obtain a low rank decomposition of any matrix $\mathbf{A}$ as $\mathbf{A} \approx \mathbf{L}\mathbf{R}$, where $\mathbf{L}$ and $\mathbf{R}$ are the low rank factors. The total number of elements in $\mathbf{L}$ and $\mathbf{R}$ can be significantly less than that in $\mathbf{A}$. Furthermore, the entries of $\mathbf{L}$ and $\mathbf{R}$ are quantized to low precision formats $--$ compressing $\mathbf{A}$ by giving us a low rank and low precision factorization. Our algorithm first computes an approximate basis of the range space of $\mathbf{A}$ by randomly sketching its columns, followed by a quantization of the vectors constituting this basis. It then computes approximate projections of the columns of $\mathbf{A}$ onto this quantized basis. We derive upper bounds on the approximation error of our algorithm, and analyze the impact of target rank and quantization bit-budget. The tradeoff between compression ratio and approximation accuracy allows for flexibility in choosing these parameters based on specific application requirements. We empirically demonstrate the efficacy of our algorithm in image compression, nearest neighbor classification of image and text embeddings, and compressing the layers of LlaMa-$7$b. Our results illustrate that we can achieve compression ratios as aggressive as one bit per matrix coordinate, all while surpassing or maintaining the performance of traditional compression techniques.

The Computation of Low Multilinear Rank Approximations of Tensors Via Power Scheme and Random Projection

Randomized Algorithms for the Low Multilinear Rank Approximations of Tensors

Randomized Algorithms for the Computation of Multilinear Rank-$$(\Mu _1,\mu _2,\mu _3)$$ Approximations

Efficient algorithms for Tucker decomposition via approximate matrix multiplication

Randomized Algorithms for the Approximations of Tucker and the Tensor Train Decompositions.

An Efficient Randomized Algorithm for Computing the Approximate Tucker Decomposition

A Random Matrix Approach to Low-Multilinear-Rank Tensor Approximation

Randomized algorithms for computing the tensor train approximation and their applications

Nonconvex Robust High-Order Tensor Completion Using Randomized Low-Rank Approximation

Efficient randomized tensor-based algorithms for function approximation and low-rank kernel interactions

On the Global Convergence of the Alternating Least Squares Method for Rank-One Approximation to Generic Tensors

Matrix Compression via Randomized Low Rank and Low Precision Factorization

Efficient Alternating Least Squares Algorithms for Low Multilinear Rank Approximation of Tensors

Efficient Algorithms for Low Tubal Rank Tensor Approximation with Applications to Image Compression, Super-Resolution and Deep Learning

Compressed Randomized UTV Decompositions for Low-Rank Approximations and Big Data Applications

Robust Approximations of Low-Rank Minimization for Tensor Completion

Low Rank Tensor Decompositions and Approximations

Riemannian Preconditioned Coordinate Descent for Low Multilinear Rank Approximation

Fast Randomized Tensor Singular Value Thresholding for Low‐rank Tensor Optimization

A sequential multilinear Nyström algorithm for streaming low-rank approximation of tensors in Tucker format

Multilinear Low-Rank Tensors on Graphs & Applications