Abstract:We study the problem of regression in a generalized linear model (GLM) with multiple signals and latent variables. This model, which we call a matrix GLM, covers many widely studied problems in statistical learning, including mixed linear regression, max-affine regression, and mixture-of-experts. In mixed linear regression, each observation comes from one of $L$ signal vectors (regressors), but we do not know which one; in max-affine regression, each observation comes from the maximum of $L$ affine functions, each defined via a different signal vector. The goal in all these problems is to estimate the signals, and possibly some of the latent variables, from the observations. We propose a novel approximate message passing (AMP) algorithm for estimation in a matrix GLM and rigorously characterize its performance in the high-dimensional limit. This characterization is in terms of a state evolution recursion, which allows us to precisely compute performance measures such as the asymptotic mean-squared error. The state evolution characterization can be used to tailor the AMP algorithm to take advantage of any structural information known about the signals. Using state evolution, we derive an optimal choice of AMP `denoising' functions that minimizes the estimation error in each iteration. The theoretical results are validated by numerical simulations for mixed linear regression, max-affine regression, and mixture-of-experts. For max-affine regression, we propose an algorithm that combines AMP with expectation-maximization to estimate intercepts of the model along with the signals. The numerical results show that AMP significantly outperforms other estimators for mixed linear regression and max-affine regression in most parameter regimes.

Message Passing with L1 Penalized KL Minimization

An Expectation Propagation Perspective on Approximate Message Passing

Message Passing with Relaxed Moment Matching

Unitary Approximate Message Passing for Sparse Bayesian Learning

Concise Derivation of Complex Bayesian Approximate Message Passing Via Expectation Propagation.

Generalizing Expectation Propagation with Mixtures of Exponential Family Distributions and an Application to Bayesian Logistic Regression.

Approximate Posterior Inference for Bayesian Models: Black-Box Expectation Propagation

Unifying Message Passing Algorithms under the Framework of Constrained Bethe Free Energy Minimization

Concise Derivation for Generalized Approximate Message Passing Using Expectation Propagation

Deep learning via message passing algorithms based on belief propagation

Mixed Regression via Approximate Message Passing

A Concise Tutorial on Approximate Message Passing

Understanding belief propagation and its generalizations

Probabilistic Inference of Bayesian Neural Networks with Generalized Expectation Propagation

Convergence Rates of Smooth Message Passing with Rounding in Entropy-Regularized MAP Inference

Low complexity sparse Bayesian learning using combined belief propagation and mean field with a stretched factor graph.

Computationally Efficient Sparse Bayesian Learning Via Generalized Approximate Message Passing

Expectation Propagation for approximate Bayesian inference

Scalable marginalization of correlated latent variables with applications to learning particle interaction kernels

Approximate Inference and Constrained Optimization

Kernel-Based Just-In-Time Learning for Passing Expectation Propagation Messages