Abstract:Recently, many machine learning and statistical models such as non-linear regressions, the Single Index, Multi-index, Varying Coefficient Index Models and Two-layer Neural Networks can be reduced to or be seen as a special case of a new model which is called the Stochastic Linear Combination of Non-linear Regressions model. However, due to the high non-convexity of the problem, there is no previous work study how to estimate the model. In this paper, we provide the first study on how to estimate the model efficiently and scalably. Specifically, we first show that with some mild assumptions, if the variate vector x is multivariate Gaussian, then there is an algorithm whose output vectors have l(2)-norm estimation errors of O(root p/n) with high probability, where p is the dimension of x and n is the number of samples. The key idea of the proof is based on an observation motived by the Stein's lemma. Then we extend our result to the case where x is bounded and sub-Gaussian using the zero-bias transformation, which could be seen as a generalization of the classic Stein's lemma. We also show that with some additional assumptions there is an algorithm whose output vectors have l(infinity)-norm estimation errors of O(1 root p + root p/n) with high probability. We also provide a concrete example to show that there exists some link function which satisfies the previous assumptions. Finally, for both Gaussian and sub-Gaussian cases we propose a faster sub-sampling based algorithm and show that when the sub-sample sizes are large enough then the estimation errors will not be sacrificed by too much. Experiments for both cases support our theoretical results. To the best of our knowledge, this is the first work that studies and provides theoretical guarantees for the stochastic linear combination of non-linear regressions model. (C) 2020 Elsevier B.V. All rights reserved.

A new analysis of optimal estimation and prediction under linear mixed models

Best linear unbiased prediction for linear combinations in general mixed linear models

Statistical inference in partially time-varying coefficient models

On a Biased Prediction Based on Optimal Mean Square Error Criterion

Valid predictions of random quantities in linear mixed models

A penalized likelihood approach for efficiently estimating a partially linear additive transformation model with current status data

Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

Mixed Linear Model Approaches for Analyzing Genetic Models of Complex Quantitative Traits

Estimating individual treatment effect: generalization bounds and algorithms

Electronic structure study by means of X-ray spectroscopy and theoretical calculations of the "ferric star" single molecule magnet

Essential formulae for restricted maximum likelihood and its derivatives associated with the linear mixed models

A Nonprobabilistic Approach for Linear Model Selection

Estimating Stochastic Linear Combination of Non-Linear Regressions Efficiently and Scalably

Optimal designs for linear models with Fredholm-type errors

Partial Linear Model Averaging Prediction for Longitudinal Data

Functional Linear Regression with Mixed Predictors

Sublinear expectation linear regression

M-estimation for linear models with exchangeable errors

Tensor Graphical Model: Non-Convex Optimization and Statistical Inference

Mlinear: Rethink the Linear Model for Time-series Forecasting

Optimal Model Averaging Estimation for Generalized Linear Models and Generalized Linear Mixed-Effects Models