Abstract:Factorizing low-rank matrices is a problem with many applications in machine learning and statistics, ranging from sparse PCA to community detection and sub-matrix localization. For probabilistic models in the Bayes optimal setting, general expressions for the mutual information have been proposed using powerful heuristic statistical physics computations via the replica and cavity methods, and proven in few specific cases by a variety of methods. Here, we use the spatial coupling methodology developed in the framework of error correcting codes, to rigorously derive the mutual information for the symmetric rank-one case. We characterize the detectability phase transitions in a large set of estimation problems, where we show that there exists a gap between what currently known polynomial algorithms (in particular spectral methods and approximate message-passing) can do and what is expected information theoretically. Moreover, we show that the computational gap vanishes for the proposed spatially coupled model, a promising feature with many possible applications. Our proof technique has an interest on its own and exploits three essential ingredients: the interpolation method first introduced in statistical physics, the analysis of approximate message-passing algorithms first introduced in compressive sensing, and the theory of threshold saturation for spatially coupled systems first developed in coding theory. Our approach is very generic and can be applied to many other open problems in statistical estimation where heuristic statistical physics predictions are available.

A low-rank spectral method for learning Markov models

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Low-rank matrix estimation via nonconvex spectral regularized methods in errors-in-variables matrix regression

Adaptive Low-Nonnegative-Rank Approximation for State Aggregation of Markov Chains

Rank-Constrained Least-Squares: Prediction and Inference

Low-Rank Tensors for Multi-Dimensional Markov Models

Non-asymptotic Estimates for Markov Transition Matrices with Rigorous Error Bounds

Low-rank matrix estimation in multi-response regression with measurement errors: Statistical and computational guarantees.

Adaptive Penalized Likelihood method for Markov Chains

Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimation

Can Learning Be Explained By Local Optimality In Low-rank Matrix Recovery?

Low-rank matrix recovery with non-quadratic loss: projected gradient method and regularity projection oracle

Structured Matrix Learning under Arbitrary Entrywise Dependence and Estimation of Markov Transition Kernel

Low-rank structure learning via nonconvex heuristic recovery.

Low-rank matrix estimation via nonconvex optimization methods in multi-response errors-in-variables regression

Bayesian Learning for Low-Rank matrix reconstruction

Estimation of high-dimensional low-rank matrices

Rank-one matrix estimation: analysis of algorithmic and information theoretic limits by the spatial coupling method

Quantized Low-Rank Multivariate Regression with Random Dithering

Efficient Low-Rank Matrix Estimation, Experimental Design, and Arm-Set-Dependent Low-Rank Bandits

Low-Rank Constraints for Fast Inference in Structured Models