Abstract:Stochastic Gradient Langevin Dynamics (SGLD) have been widely used for Bayesian sampling from certain probability distributions, incorporating derivatives of the log-posterior. With the derivative evaluation of the log-posterior distribution, SGLD methods generate samples from the distribution through performing as a thermostats dynamics that traverses over gradient flows of the log-posterior with certainly controllable perturbation. Even when the density is not known, existing solutions still can first learn the kernel density models from the given datasets, then produce new samples using the SGLD over the kernel density derivatives. In this work, instead of exploring new samples from kernel spaces, a novel SGLD sampler, namely, Randomized Measurement Langevin Dynamics (RMLD) is proposed to sample the high-dimensional sparse representations from the spectral domain of a given dataset. Specifically, given a random measurement matrix for sparse coding, RMLD first derives a novel likelihood evaluator of the probability distribution from the loss function of LASSO, then samples from the high-dimensional distribution using stochastic Langevin dynamics with derivatives of the logarithm likelihood and Metropolis–Hastings sampling. In addition, new samples in low-dimensional measuring spaces can be regenerated using the sampled high-dimensional vectors and the measurement matrix. The algorithm analysis shows that RMLD indeed projects a given dataset into a high-dimensional Gaussian distribution with Laplacian prior, then draw new sparse representation from the dataset through performing SGLD over the distribution. Extensive experiments have been conducted to evaluate the proposed algorithm using real-world datasets. The performance comparisons on three real-world applications demonstrate the superior performance of RMLD beyond baseline methods.

Least Squares Approximation Via Sparse Subsampled Randomized Hadamard Transform

Effective Dimension Adaptive Sketching Methods for Faster Regularized Least-Squares Optimization

Optimal Randomized First-Order Methods for Least-Squares Problems

Randomized algorithms for total least squares problems.

Approximating Element-Wise Functions of Matrix with Improved Streaming Randomized SVD

Randomised subspace methods for non-convex optimization, with applications to nonlinear least-squares

Randomized Alternating Least Squares for Canonical Tensor Decompositions: Application to a PDE with Random Data

Randomized least-squares with minimal oversampling and interpolation in general spaces

New algorithms for computing the least trimmed squares estimator

An adaptive shortest-solution guided decimation approach to sparse high-dimensional linear regression

Sample size determination for multidimensional parameters and the A-optimal subsampling in a big data linear regression model

Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squares

Randomized Iterative Hard Thresholding for Sparse Approximations

High Dimensional Robust Sparse Regression

Efficient Bounds and Estimates for Canonical Angles in Randomized Subspace Approximations

Random Angular Projection for Fast Nearest Subspace Search

Parallel Sparse Subspace Clustering Via Joint Sample and Parameter Blockwise Partition.

Sampling Sparse Representations with Randomized Measurement Langevin Dynamics

Theoretical and Experimental Analysis of a Randomized Algorithm for Sparse Fourier Transform Analysis

Low Rank Approximation and Regression in Input Sparsity Time