Abstract:Recently, many machine learning and statistical models such as non-linear regressions, the Single Index, Multi-index, Varying Coefficient Index Models and Two-layer Neural Networks can be reduced to or be seen as a special case of a new model which is called the Stochastic Linear Combination of Non-linear Regressions model. However, due to the high non-convexity of the problem, there is no previous work study how to estimate the model. In this paper, we provide the first study on how to estimate the model efficiently and scalably. Specifically, we first show that with some mild assumptions, if the variate vector x is multivariate Gaussian, then there is an algorithm whose output vectors have l(2)-norm estimation errors of O(root p/n) with high probability, where p is the dimension of x and n is the number of samples. The key idea of the proof is based on an observation motived by the Stein's lemma. Then we extend our result to the case where x is bounded and sub-Gaussian using the zero-bias transformation, which could be seen as a generalization of the classic Stein's lemma. We also show that with some additional assumptions there is an algorithm whose output vectors have l(infinity)-norm estimation errors of O(1 root p + root p/n) with high probability. We also provide a concrete example to show that there exists some link function which satisfies the previous assumptions. Finally, for both Gaussian and sub-Gaussian cases we propose a faster sub-sampling based algorithm and show that when the sub-sample sizes are large enough then the estimation errors will not be sacrificed by too much. Experiments for both cases support our theoretical results. To the best of our knowledge, this is the first work that studies and provides theoretical guarantees for the stochastic linear combination of non-linear regressions model. (C) 2020 Elsevier B.V. All rights reserved.

Estimating the Ratio of Two Functions in a Nonparametric Regression Model

Estimating Stochastic Linear Combination of Non-Linear Regressions

Estimating nonlinear regression errors without doing regression

Nonparametric Inference about Mean Functionals of Nonignorable Nonresponse Data Without Identifying the Joint Distribution

Estimating Stochastic Linear Combination of Non-Linear Regressions Efficiently and Scalably

Optimal Estimation of Derivatives in Nonparametric Regression

The Consistency for Estimator of Nonparametric Regression Model Based on NOD Errors

Ratio tests for variance change in nonparametric regression

A joint estimation approach for monotonic regression functions in general dimensions

NONPARAMETRIC REGRESSION FUNCTION ESTIMATION BASED ON CENSORED DATA

Stable and bias-corrected estimation for nonparametric regression models

Simulation-based two-stage estimation for multiple nonparametric regression

An Adaptive Estimation for Covariate-Adjusted Nonparametric Regression Model

Two-Stage Estimation for Seemingly Unrelated Nonparametric Regression Models

Nonparametric estimation of the log odds ratio for sparse data by kernel smoothing

Ratio Test to Detect Variance Change in Nonparametric Regression Model under Random Design

Nonparametric Estimation via Partial Derivatives

Efficient Estimation for Error Component Seemingly Unrelated Nonparametric Regression Models

Difference-based estimation for error variances in repeated measurement regression models

The Asymptotic Normality of the Linear Weighted Estimator in Nonparametric Regression Models

Estimation and Model Selection for Nonparametric Function-on-Function Regression