Abstract:We study the task of learning latent-variable models. An obstacle towards designing efficient algorithms for such models is the necessity of approximating moment tensors of super-constant degree. Motivated by such applications, we develop a general efficient algorithm for implicit moment tensor computation. Our algorithm computes in $\mathrm{poly}(d, k)$ time a succinct approximate description of tensors of the form $M_m=\sum_{i=1}^{k}w_iv_i^{\otimes m}$, for $w_i\in\mathbb{R}_+$--even for $m=\omega(1)$--assuming there exists a polynomial-size arithmetic circuit whose expected output on an appropriate samplable distribution is equal to $M_m$, and whose covariance on this input is bounded. Our framework broadly generalizes the work of~\cite{LL21-opt} which developed an efficient algorithm for the specific moment tensors that arise in clustering mixtures of spherical Gaussians. By leveraging our general algorithm, we obtain the first polynomial-time learners for the following models. * Mixtures of Linear Regressions. We give a $\mathrm{poly}(d, k, 1/\epsilon)$-time algorithm for this task. The previously best algorithm has super-polynomial complexity in $k$. * Learning Mixtures of Spherical Gaussians. We give a $\mathrm{poly}(d, k, 1/\epsilon)$-time density estimation algorithm, under the condition that the means lie in a ball of radius $O(\sqrt{\log k})$. Prior algorithms incur super-polynomial complexity in $k$. We also give a $\mathrm{poly}(d, k, 1/\epsilon)$-time parameter estimation algorithm, under the {\em optimal} mean separation of $\Omega(\log^{1/2}(k/\epsilon))$. * PAC Learning Sums of ReLUs. We give a learner with complexity $\mathrm{poly}(d, k) 2^{\mathrm{poly}(1/\epsilon)}$. This is the first algorithm for this task that runs in $\mathrm{poly}(d, k)$ time for subconstant values of $\epsilon = o_{k, d}(1)$.

Gaussian Mixture Identifiability from degree 6 Moments

Estimating Gaussian mixtures using sparse polynomial moment systems

Moment varieties of the inverse Gaussian and gamma distributions are nondefective

Generalized Identifiability Bounds for Mixture Models with Grouped Samples

Moment varieties from inverse Gaussian and gamma distributions

Diagonal Gaussian Mixture Models and Higher Order Tensor Decompositions

Identifiability of multivariate logistic mixture models

On the entropy and information of Gaussian mixtures

Entropic characterization of optimal rates for learning Gaussian mixtures

The Breakdown of Gaussian Universality in Classification of High-dimensional Mixtures

Interpolating Discriminant Functions in High-Dimensional Gaussian Latent Mixtures

Statistical Inference in Classification of High-dimensional Gaussian Mixture

Universality laws for Gaussian mixtures in generalized linear models*

Learning general Gaussian mixtures with efficient score matching

Implicit High-Order Moment Tensor Estimation and Learning Latent Variable Models

Approximate independence of permutation mixtures

Mixture models for simultaneous classification and reduction of three-way data

Identifiability of overcomplete independent component analysis

Confirmation Bias in Gaussian Mixture Models

Generalized identifiability of sums of squares