Abstract:Recently, many machine learning and statistical models such as non-linear regressions, the Single Index, Multi-index, Varying Coefficient Index Models and Two-layer Neural Networks can be reduced to or be seen as a special case of a new model which is called the Stochastic Linear Combination of Non-linear Regressions model. However, due to the high non-convexity of the problem, there is no previous work study how to estimate the model. In this paper, we provide the first study on how to estimate the model efficiently and scalably. Specifically, we first show that with some mild assumptions, if the variate vector x is multivariate Gaussian, then there is an algorithm whose output vectors have l(2)-norm estimation errors of O(root p/n) with high probability, where p is the dimension of x and n is the number of samples. The key idea of the proof is based on an observation motived by the Stein's lemma. Then we extend our result to the case where x is bounded and sub-Gaussian using the zero-bias transformation, which could be seen as a generalization of the classic Stein's lemma. We also show that with some additional assumptions there is an algorithm whose output vectors have l(infinity)-norm estimation errors of O(1 root p + root p/n) with high probability. We also provide a concrete example to show that there exists some link function which satisfies the previous assumptions. Finally, for both Gaussian and sub-Gaussian cases we propose a faster sub-sampling based algorithm and show that when the sub-sample sizes are large enough then the estimation errors will not be sacrificed by too much. Experiments for both cases support our theoretical results. To the best of our knowledge, this is the first work that studies and provides theoretical guarantees for the stochastic linear combination of non-linear regressions model. (C) 2020 Elsevier B.V. All rights reserved.

Statistical Inference of Seemingly Unrelated Partially Linear Additive Regression Models

Statistical inference on seemingly unrelated non-parametric regression models with serially correlated errors

Weighted Profile Least Squares Estimation for a Panel Data Varying-Coefficient Partially Linear Model

Estimation and model selection in generalized additive partial linear models for correlated data with diverging number of covariates

Efficient Estimation for Error Component Seemingly Unrelated Nonparametric Regression Models

PENALIZED VARIABLE SELECTION PROCEDURE FOR COX MODELS WITH SEMIPARAMETRIC RELATIVE RISK

Bias-Corrected Inference of High-Dimensional Generalized Linear Models

Empirical-Likelihood-Based Inference for Partially Linear Models

Generalized Partially Linear Models for Incomplete Longitudinal Data in the Presence of Population-Level Information

A penalized likelihood approach for efficiently estimating a partially linear additive transformation model with current status data

Additive model building for spatial regression

Semiparametric Inference for Estimating Equations with Nonignorably Missing Covariates

A robust estimation based on penalised regularisation for the varying-coefficient additive model

Statistical Inference for Partially Linear Regression Models with Measurement Errors

Estimating Stochastic Linear Combination of Non-Linear Regressions Efficiently and Scalably

Double robust semiparametric weighted M-estimators of a zero-inflated Poisson regression with missing data in covariates

Estimation of Semi-Parametric Additive Coefficient Model

Large System Of Seemingly Unrelated Regressions: A Penalized Quasi-Maximum Likelihood Estimation Perspective

Statistical inference in functional semiparametric spatial autoregressive model

Partial Linear Model Averaging Prediction for Longitudinal Data

Doubly robust and efficient estimators for heteroscedastic partially linear single-index models allowing high dimensional covariates