Abstract:Recently, many machine learning and statistical models such as non-linear regressions, the Single Index, Multi-index, Varying Coefficient Index Models and Two-layer Neural Networks can be reduced to or be seen as a special case of a new model which is called the Stochastic Linear Combination of Non-linear Regressions model. However, due to the high non-convexity of the problem, there is no previous work study how to estimate the model. In this paper, we provide the first study on how to estimate the model efficiently and scalably. Specifically, we first show that with some mild assumptions, if the variate vector x is multivariate Gaussian, then there is an algorithm whose output vectors have l(2)-norm estimation errors of O(root p/n) with high probability, where p is the dimension of x and n is the number of samples. The key idea of the proof is based on an observation motived by the Stein's lemma. Then we extend our result to the case where x is bounded and sub-Gaussian using the zero-bias transformation, which could be seen as a generalization of the classic Stein's lemma. We also show that with some additional assumptions there is an algorithm whose output vectors have l(infinity)-norm estimation errors of O(1 root p + root p/n) with high probability. We also provide a concrete example to show that there exists some link function which satisfies the previous assumptions. Finally, for both Gaussian and sub-Gaussian cases we propose a faster sub-sampling based algorithm and show that when the sub-sample sizes are large enough then the estimation errors will not be sacrificed by too much. Experiments for both cases support our theoretical results. To the best of our knowledge, this is the first work that studies and provides theoretical guarantees for the stochastic linear combination of non-linear regressions model. (C) 2020 Elsevier B.V. All rights reserved.

Unsupervised Liu-type shrinkage estimators for mixture of regression models

Liu-type Shrinkage Estimators for Mixture of Logistic Regressions: An Osteoporosis Study

A permutation-based approach to inference for weighted sum regression with correlated chemical mixtures

On Mixture Regression Shrinkage and Selection via the MR-LASSO

Sequential Estimation for Mixture of Regression Models for Heterogeneous Population

Model selection in finite mixture of regression models: a Bayesian approach with innovative weighted g priors and reversible jump Markov chain Monte Carlo implementation (vol 85, pg 2456, 2015)

Semiparametric Mixture Regression with Unspecified Error Distributions

A Nonparametric Estimation Method for the Multivariate Mixture Models

Leveraging independence in high-dimensional mixed linear regression

Investigating Approaches to Estimating Covariate Effects in Growth Mixture Modeling: A Simulation Study

Multiple observers ranked set samples for shrinkage estimators

A penalized likelihood approach for efficiently estimating a partially linear additive transformation model with current status data

Model Selection for Exponential Power Mixture Regression Models

Expectation-Maximization-Maximization: A Feasible MLE Algorithm for the Three-Parameter Logistic Model Based on a Mixture Modeling Reformulation

A globally convergent algorithm for lasso-penalized mixture of linear regression models

Statistical inference for a mixture of Markov jump processes

Maximum-penalized-likelihood estimation for independent and Markov-dependent mixture models.

Liu-type Negative Binomial Regression: A Comparison of Recent Estimators and Applications

Estimating Stochastic Linear Combination of Non-Linear Regressions Efficiently and Scalably

Unsupervised learning of regression mixture models with unknown number of components

Families of Parsimonious Finite Mixtures of Regression Models