Abstract:We consider the constrained sampling problem where the goal is to sample from a target distribution $\pi(x)\propto e^{-f(x)}$ when $x$ is constrained to lie on a convex body $\mathcal{C}$. Motivated by penalty methods from continuous optimization, we propose penalized Langevin Dynamics (PLD) and penalized underdamped Langevin Monte Carlo (PULMC) methods that convert the constrained sampling problem into an unconstrained sampling problem by introducing a penalty function for constraint violations. When $f$ is smooth and gradients are available, we get $\tilde{\mathcal{O}}(d/\varepsilon^{10})$ iteration complexity for PLD to sample the target up to an $\varepsilon$-error where the error is measured in the TV distance and $\tilde{\mathcal{O}}(\cdot)$ hides logarithmic factors. For PULMC, we improve the result to $\tilde{\mathcal{O}}(\sqrt{d}/\varepsilon^{7})$ when the Hessian of $f$ is Lipschitz and the boundary of $\mathcal{C}$ is sufficiently smooth. To our knowledge, these are the first convergence results for underdamped Langevin Monte Carlo methods in the constrained sampling that handle non-convex $f$ and provide guarantees with the best dimension dependency among existing methods with deterministic gradient. If unbiased stochastic estimates of the gradient of $f$ are available, we propose PSGLD and PSGULMC methods that can handle stochastic gradients and are scaleable to large datasets without requiring Metropolis-Hasting correction steps. For PSGLD and PSGULMC, when $f$ is strongly convex and smooth, we obtain $\tilde{\mathcal{O}}(d/\varepsilon^{18})$ and $\tilde{\mathcal{O}}(d\sqrt{d}/\varepsilon^{39})$ iteration complexity in W2 distance. When $f$ is smooth and can be non-convex, we provide finite-time performance bounds and iteration complexity results. Finally, we illustrate the performance on Bayesian LASSO regression and Bayesian constrained deep learning problems.

Concentration Estimates for Learning with Unbounded Sampling

Learning with coefficient-based regularization and ℓ 1 −penalty

Concentration Estimates for Learning with ℓ1-Regularizer and Data Dependent Hypothesis Spaces

Online Regression with Unbounded Sampling.

Classification with Non-I.i.d. Sampling

Learning Theory Estimates for Coefficient-Based Regularized Regression

ERM Learning with Unbounded Sampling

Convergence of Unregularized Online Learning Algorithms

Quantile Regression With L(1)-Regularization And Gaussian Kernels

On the Concentration of the Minimizers of Empirical Risks

Learning Rates of Least-Square Regularized Regression with Polynomial Kernels

Random sampling and reconstruction of concentrated signals in a reproducing kernel space

Learning rates for regularized least squares ranking algorithm

Learning Rate Free Sampling in Constrained Domains

Penalized Overdamped and Underdamped Langevin Monte Carlo Algorithms for Constrained Sampling

Local convergence rates of the nonparametric least squares estimator with applications to transfer learning

Achieving High Convergence Rates by Quasi-Monte Carlo and Importance Sampling for Unbounded Integrands

Concentration Inequalities for Statistical Inference

Posterior concentration and fast convergence rates for generalized Bayesian learning

Analysis of Least Squares Regularized Regression in Reproducing Kernel Krein Spaces

Efficient Concentration with Gaussian Approximation