Abstract:Approximate message passing (AMP) has emerged both as a popular class of iterative algorithms and as a powerful analytic tool in a wide range of statistical estimation problems and statistical physics models. A well established line of AMP theory proves Gaussian approximations for the empirical distributions of the AMP iterate in the high dimensional limit, under the GOE random matrix model and its variants. This paper provides a non-asymptotic, leave-one-out representation for the AMP iterate that holds under a broad class of Gaussian random matrix models with general variance profiles. In contrast to the typical AMP theory that describes the empirical distributions of the AMP iterate via a low dimensional state evolution, our leave-one-out representation yields an intrinsically high dimensional state evolution formula which provides non-asymptotic characterizations for the possibly heterogeneous, entrywise behavior of the AMP iterate under the prescribed random matrix models. To exemplify some distinct features of our AMP theory in applications, we analyze, in the context of regularized linear estimation, the precise stochastic behavior of the Ridge estimator for independent and non-identically distributed observations whose covariates exhibit general variance profiles. We find that its finite-sample distribution is characterized via a weighted Ridge estimator in a heterogeneous Gaussian sequence model. Notably, in contrast to the i.i.d. sampling scenario, the effective noise and regularization are now full dimensional vectors determined via a high dimensional system of equations. Our leave-one-out method of proof differs significantly from the widely adopted conditioning approach for rotational invariant ensembles, and relies instead on an inductive method that utilizes almost solely integration-by-parts and concentration techniques.

Local approximate inference algorithms

Bethe Bounds and Approximating the Global Optimum

Maximum a posteriori estimation in graphical models using local linear approximation

Approximating the Bethe partition function

Approximating the Sum Operation for Marginal-MAP Inference.

Rate-optimal refinement strategies for local approximation MCMC

l_inf-approximation of localized distributions

Inferring Change Points in High-Dimensional Regression via Approximate Message Passing

Variational Algorithms for Marginal MAP

Spectral Approximate Inference

A leave-one-out approach to approximate message passing

Convergence Rates of Smooth Message Passing with Rounding in Entropy-Regularized MAP Inference

Neural Network Approximators for Marginal MAP in Probabilistic Circuits

Approximate labeling via graph cuts based on linear programming

Lagrangian-based methods for finding MAP solutions for MRF models

Optimal approximating Markov chains for Bayesian inference

Convergence Rates for the MAP of an Exponential Family and Stochastic Mirror Descent -- an Open Problem

A Non-asymptotic Analysis of Generalized Approximate Message Passing Algorithms with Right Rotationally Invariant Designs

Approximate Inference and Constrained Optimization

Efficient Localized Inference for Large Graphical Models

Amortized Bethe Free Energy Minimization for Learning MRFs