Abstract:Recently, many machine learning and statistical models such as non-linear regressions, the Single Index, Multi-index, Varying Coefficient Index Models and Two-layer Neural Networks can be reduced to or be seen as a special case of a new model which is called the Stochastic Linear Combination of Non-linear Regressions model. However, due to the high non-convexity of the problem, there is no previous work study how to estimate the model. In this paper, we provide the first study on how to estimate the model efficiently and scalably. Specifically, we first show that with some mild assumptions, if the variate vector x is multivariate Gaussian, then there is an algorithm whose output vectors have l(2)-norm estimation errors of O(root p/n) with high probability, where p is the dimension of x and n is the number of samples. The key idea of the proof is based on an observation motived by the Stein's lemma. Then we extend our result to the case where x is bounded and sub-Gaussian using the zero-bias transformation, which could be seen as a generalization of the classic Stein's lemma. We also show that with some additional assumptions there is an algorithm whose output vectors have l(infinity)-norm estimation errors of O(1 root p + root p/n) with high probability. We also provide a concrete example to show that there exists some link function which satisfies the previous assumptions. Finally, for both Gaussian and sub-Gaussian cases we propose a faster sub-sampling based algorithm and show that when the sub-sample sizes are large enough then the estimation errors will not be sacrificed by too much. Experiments for both cases support our theoretical results. To the best of our knowledge, this is the first work that studies and provides theoretical guarantees for the stochastic linear combination of non-linear regressions model. (C) 2020 Elsevier B.V. All rights reserved.

The convergence of LS estimator for nonlinear regression models based on m-WOD errors

Local Linear M‐estimation in Non‐parametric Spatial Regression

Local Linear M-Estimators in Null Recurrent Time Series

Asymptotic expansion for nonparametric M-estimator in a nonlinear regression model with long-memory errors

Variants of Non-Negative Least-Mean-Square Algorithm and Convergence Analysis

Local linear M-estimation for spatial processes in fixed-design models

Spatial local M-estimation under association

Robust estimation in a nonlinear cointegration model

Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

Statistical Inference of Seemingly Unrelated Partially Linear Additive Regression Models

WLAD-LASSO Method for Robust Estimation and Variable Selection in Partially Linear Models

On complete convergence for weighted sums of m-widely acceptable random variables under sub-linear expectations and its statistical applications

Estimating Stochastic Linear Combination of Non-Linear Regressions Efficiently and Scalably

Estimating Stochastic Linear Combination of Non-Linear Regressions

High-dimensional genome-wide association study and misspecified mixed model analysis

Weighted Stochastic Restricted Estimation in Linear Measurement Error Models

Complete convergence and complete moment convergence for weighted sums of random variables satisfying generalized Rosenthal type inequalities and an application

Asymptotics of Estimates in Constrained Nonlinear Regression with Long-Range Dependent Innovations

Weighted Lasso Estimates for Sparse Logistic Regression: Non-Asymptotic Properties with Measurement Errors

Some strong convergence properties for randomly weighted maximum partial sums of END random variables with statistical applications