Abstract:The element-wise functions of a matrix are widely used in machine learning. For the applications with large matrices, efficiently computing the matrix-vector multiplication of matrix element-wise function without explicitly constructed matrix is very desired. In this work, we aim to develop an efficient low-rank approximation of the element-wise function of matrix with the time/memory cost linear to the matrix dimension. We first propose a sparse-sign streaming randomized SVD (ssrSVD) algorithm based on a streaming singular value decomposition (SVD) algorithm and the sparse-sign random projection for the approximation of element-wise function of general asymmetric matrix. For symmetric positive semi-definite (SPSD) matrix, for which the existing Nyström [1] and FastSPSD [2] method do not perform well if the matrix's singular value decays slowly, we propose a theoretically proved shift skill to improve the approximation accuracy. Combining with the ssrSVD, we obtain the sparse-sign streaming SPSD matrix approximation with shift (S3SPSD) algorithm. Experiments are carried out to evaluate the proposed algorithms' performance in approximating element-wise functions of matrix. With the color transfer task based on the Sinkhorn algorithm, the ssrSVD algorithm largely reduces the approximation error (up to $10^{5}\times$) compared with the state-of-the-art baselines, and results in high-quality color transfer result. For the kernel matrix approximation, the proposed S3SPSD algorithm also consistently outperforms the state-of-the-art baselines. Experimental results finally validate the linear time complexity of the proposed algorithms.

Efficient Frequent Directions Algorithm for Sparse Matrices

Near Optimal Frequent Directions for Sketching Dense and Sparse Matrices

Revisiting Co-Occurring Directions: Sharper Analysis and Efficient Algorithm for Sparse Matrices.

Approximate Multiplication of Sparse Matrices with Limited Space

Co-Occuring Directions Sketching for Approximate Matrix Multiply

Relative Errors for Deterministic Low-Rank Matrix Approximations

Effective Streaming Low-Tubal-Rank Tensor Approximation via Frequent Directions

An improvement of the parameterized frequent directions algorithm

Distributed Least Squares in Small Space via Sketching and Bias Reduction

Sketching Algorithms for Sparse Dictionary Learning: PTAS and Turnstile Streaming

Optimal Matrix Sketching over Sliding Windows

An Improved Frequent Directions Algorithm for Low-Rank Approximation via Block Krylov Iteration

Efficient Matrix Sketching over Distributed Data

Near-Optimal Entrywise Sampling for Data Matrices

Frequent Direction Algorithms for Approximate Matrix Multiplication with Applications in CCA.

Diagonally-Addressed Matrix Nicknack: How to improve SpMV performance

Fast and Adaptive Sparse Precision Matrix Estimation in High Dimensions

Approximating Element-Wise Functions of Matrix with Improved Streaming Randomized SVD

Efficient Adaptive Online Learning Via Frequent Directions

Distributed-Memory Parallel Algorithms for Sparse Matrix and Sparse Tall-and-Skinny Matrix Multiplication

Fast and Accurate Log-Determinant Approximations