Abstract:Recent work in machine learning community proposed multiple methods for performing lossy compression (quantization) of large matrices. This quantization is important for accelerating matrix multiplication (main component of large language models), which is often bottlenecked by the speed of loading these matrices from memory. Unlike classical vector quantization and rate-distortion theory, the goal of these new compression algorithms is to be able to approximate not the matrices themselves, but their matrix product. Specifically, given a pair of real matrices $A,B$ an encoder (compressor) is applied to each of them independently producing descriptions with $R$ bits per entry. These representations subsequently are used by the decoder to estimate matrix product $A^\top B$. In this work, we provide a non-asymptotic lower bound on the mean squared error of this approximation (as a function of rate $R$) for the case of matrices $A,B$ with iid Gaussian entries. Algorithmically, we construct a universal quantizer based on nested lattices with an explicit guarantee of approximation error for any (non-random) pair of matrices $A$, $B$ in terms of only Frobenius norms $\|A\|_F, \|B\|_F$ and $\|A^\top B\|_F$. For iid Gaussian matrices our quantizer achieves the lower bound and is, thus, asymptotically optimal. A practical low-complexity version of our quantizer achieves performance quite close to optimal. In information-theoretic terms we derive rate-distortion function for matrix multiplication of iid Gaussian matrices.

Robust Least Squares for Quantized Data Matrices

Quantized Kernel Recursive Least Squares Algorithm.

Robust Partially-Compressed Least-Squares

Quantized Kernel Least Mean Square Algorithm

Robust State Estimation For Linear Systems With Parametric Uncertainties And Quantised Measurements

Quantile-based Random Kaczmarz for corrupted linear systems of equations

Robustness of Neural Networks to Parameter Quantization

Robust Dequantization of the Quantum Singular value Transformation and Quantum Machine Learning Algorithms

A Rigorous Analysis of Least Squares Sine Fitting Using Quantized Data: the Random Phase Case

Quantum generalized least squares method in system identification

Scalar Quantization as Sparse Least Square Optimization

Probabilistic Analysis of Least Squares, Orthogonal Projection, and QR Factorization Algorithms Subject to Gaussian Noise

Quantized Low-Rank Multivariate Regression with Random Dithering

Optimal Quantization for Matrix Multiplication

A Novel Approach to Quantized Matrix Completion Using Huber Loss Measure

Robust H∞ control with partly quantized information

On the best lattice quantizers

Sparse and Low-Rank Matrix Quantile Estimation with Application to Quadratic Regression

Data-Driven Superstabilization of Linear Systems under Quantization

RAND: Robustness Aware Norm Decay For Quantized Seq2seq Models

Towards Practical Large-scale Randomized Iterative Least Squares Solvers through Uncertainty Quantification