Abstract:Approximate message passing (AMP) has emerged both as a popular class of iterative algorithms and as a powerful analytic tool in a wide range of statistical estimation problems and statistical physics models. A well established line of AMP theory proves Gaussian approximations for the empirical distributions of the AMP iterate in the high dimensional limit, under the GOE random matrix model and its variants. This paper provides a non-asymptotic, leave-one-out representation for the AMP iterate that holds under a broad class of Gaussian random matrix models with general variance profiles. In contrast to the typical AMP theory that describes the empirical distributions of the AMP iterate via a low dimensional state evolution, our leave-one-out representation yields an intrinsically high dimensional state evolution formula which provides non-asymptotic characterizations for the possibly heterogeneous, entrywise behavior of the AMP iterate under the prescribed random matrix models. To exemplify some distinct features of our AMP theory in applications, we analyze, in the context of regularized linear estimation, the precise stochastic behavior of the Ridge estimator for independent and non-identically distributed observations whose covariates exhibit general variance profiles. We find that its finite-sample distribution is characterized via a weighted Ridge estimator in a heterogeneous Gaussian sequence model. Notably, in contrast to the i.i.d. sampling scenario, the effective noise and regularization are now full dimensional vectors determined via a high dimensional system of equations. Our leave-one-out method of proof differs significantly from the widely adopted conditioning approach for rotational invariant ensembles, and relies instead on an inductive method that utilizes almost solely integration-by-parts and concentration techniques.

Concise Derivation for Generalized Approximate Message Passing Using Expectation Propagation

An Expectation Propagation Perspective on Approximate Message Passing

Concise Derivation of Complex Bayesian Approximate Message Passing Via Expectation Propagation.

A Unitary Transform Based Generalized Approximate Message Passing

A New Insight into GAMP and AMP.

Unitary Approximate Message Passing for Sparse Bayesian Learning

An AMP-Based Low Complexity Generalized Sparse Bayesian Learning Algorithm

Bilinear Adaptive Generalized Vector Approximate Message Passing

Generalized Memory Approximate Message Passing for Generalized Linear Model.

A Concise Tutorial on Approximate Message Passing

Message Passing with L1 Penalized KL Minimization

Generalized Memory Approximate Message Passing

Generalizing Expectation Propagation with Mixtures of Exponential Family Distributions and an Application to Bayesian Logistic Regression.

Doped expectation propagation for low-complexity message passing based detection

Message Passing with Relaxed Moment Matching

A leave-one-out approach to approximate message passing

A Matching Pursuit Generalized Approximate Message Passing Algorithm.

Multi-Layer Bilinear Generalized Approximate Message Passing

Unifying Message Passing Algorithms under the Framework of Constrained Bethe Free Energy Minimization

Computationally Efficient Sparse Bayesian Learning Via Generalized Approximate Message Passing

Generalized Approximate Survey Propagation for High-Dimensional Estimation