Abstract:Recently, many machine learning and statistical models such as non-linear regressions, the Single Index, Multi-index, Varying Coefficient Index Models and Two-layer Neural Networks can be reduced to or be seen as a special case of a new model which is called the Stochastic Linear Combination of Non-linear Regressions model. However, due to the high non-convexity of the problem, there is no previous work study how to estimate the model. In this paper, we provide the first study on how to estimate the model efficiently and scalably. Specifically, we first show that with some mild assumptions, if the variate vector x is multivariate Gaussian, then there is an algorithm whose output vectors have l(2)-norm estimation errors of O(root p/n) with high probability, where p is the dimension of x and n is the number of samples. The key idea of the proof is based on an observation motived by the Stein's lemma. Then we extend our result to the case where x is bounded and sub-Gaussian using the zero-bias transformation, which could be seen as a generalization of the classic Stein's lemma. We also show that with some additional assumptions there is an algorithm whose output vectors have l(infinity)-norm estimation errors of O(1 root p + root p/n) with high probability. We also provide a concrete example to show that there exists some link function which satisfies the previous assumptions. Finally, for both Gaussian and sub-Gaussian cases we propose a faster sub-sampling based algorithm and show that when the sub-sample sizes are large enough then the estimation errors will not be sacrificed by too much. Experiments for both cases support our theoretical results. To the best of our knowledge, this is the first work that studies and provides theoretical guarantees for the stochastic linear combination of non-linear regressions model. (C) 2020 Elsevier B.V. All rights reserved.

A High Accuracy Stochastic Estimation of a Nonlinear Deterministic Model

Nonlinear Recursive Minimum Model Error Estimation

Resilient Asynchronous State Estimation for Markovian Jump Neural Networks Subject to Stochastic Nonlinearities and Sensor Saturations

A Multilevel Approach for Stochastic Nonlinear Optimal Control

Weighted Stochastic Restricted Estimation in Linear Measurement Error Models

Probabilistic error estimation for non-intrusive reduced models learned from data of systems governed by linear parabolic partial differential equations

Estimating Stochastic Linear Combination of Non-Linear Regressions Efficiently and Scalably

A Stochastic Newton Method for Nonlinear Equations

Predicting the Output From a Stochastic Computer Model When a Deterministic Approximation is Available

Stochastic state estimation via incremental iterative sparse polynomial chaos based Bayesian-Gauss-Newton-Markov-Kalman filter

Adaptive Estimation Algorithm for Systems with Multiplicative Noise and Unknown Transfer Matrix

On the Robust Estimation of Small Failure Probabilities for Strong Nonlinear Models

Stochastic Variational Bayesian Inference for a Nonlinear Forward Model

Markov Chain Variance Estimation: A Stochastic Approximation Approach

Estimating Stochastic Linear Combination of Non-Linear Regressions

Formal Verification of Unknown Stochastic Systems via Non-parametric Estimation

State estimation of a physical system with unknown governing equations

Estimation of Dynamical Systems in Noisy Conditions and with Constraints

The stochastic approximation method for the estimation of a multivariate probability density

Estimation for the Linear Model with Uncertain Covariance Matrices

A Bayesian Stochastic Approximation Method