Abstract:Generalization theory has been established for sparse deep neural networks under high-dimensional regime. Beyond generalization, parameter estimation is also important since it is crucial for variable selection and interpretability of deep neural networks. Current theoretical studies concerning parameter estimation mainly focus on two-layer neural networks, which is due to the fact that the convergence of parameter estimation heavily relies on the regularity of the Hessian matrix, while the Hessian matrix of deep neural networks is highly singular. To avoid the unidentifiability of deep neural networks in parameter estimation, we propose to conduct nonparametric estimation of partial derivatives with respect to inputs. We first show that model convergence of sparse deep neural networks is guaranteed in that the sample complexity only grows with the logarithm of the number of parameters or the input dimension when the $\ell_{1}$-norm of parameters is well constrained. Then by bounding the norm and the divergence of partial derivatives, we establish that the convergence rate of nonparametric estimation of partial derivatives scales as $\mathcal{O}(n^{-1/4})$, a rate which is slower than the model convergence rate $\mathcal{O}(n^{-1/2})$. To the best of our knowledge, this study combines nonparametric estimation and parametric sparse deep neural networks for the first time. As nonparametric estimation of partial derivatives is of great significance for nonlinear variable selection, the current results show the promising future for the interpretability of deep neural networks.

Estimation of Linear Functionals in High-Dimensional Linear Models: From Sparsity to Nonsparsity

Sparse Semiparametric Efficient Estimation in High-Dimensional Linear Regression Models

Estimation and Inference for High-Dimensional Non-Sparse Models

Sparse Nonlinear Regression: Parameter Estimation and Asymptotic Inference

Partially Functional Linear Regression in High Dimensions

Factor-guided estimation of large covariance matrix function with conditional functional sparsity

Variable Selection and Minimax Prediction in High-dimensional Functional Linear Model

A Constructive Approach to High-dimensional Regression

Functional Linear Regression with Mixed Predictors

An Iterative Algorithm for High-Dimensional Linear Models with Both Sparse and Non-Sparse Structures

A minimax optimal approach to high-dimensional double sparse linear regression

Inference for High-Dimensional Sparse Econometric Models

Smooth and Locally Sparse Estimation for Multiple-Output Functional Linear Regression

Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression

On Sparse Estimation for Semiparametric Linear Transformation Models

Error Density Estimation in High-Dimensional Sparse Linear Model

Semiparametric efficient estimation in high‐dimensional partial linear regression models

High-Dimensional Linear Regression via Implicit Regularization

Kernel-based Estimation for Partially Functional Linear Model: Minimax Rates and Randomized Sketches

Confidence regions for high-dimensional generalized linear models under sparsity

High Dimensional Robust Sparse Regression