Abstract:Generalization theory has been established for sparse deep neural networks under high-dimensional regime. Beyond generalization, parameter estimation is also important since it is crucial for variable selection and interpretability of deep neural networks. Current theoretical studies concerning parameter estimation mainly focus on two-layer neural networks, which is due to the fact that the convergence of parameter estimation heavily relies on the regularity of the Hessian matrix, while the Hessian matrix of deep neural networks is highly singular. To avoid the unidentifiability of deep neural networks in parameter estimation, we propose to conduct nonparametric estimation of partial derivatives with respect to inputs. We first show that model convergence of sparse deep neural networks is guaranteed in that the sample complexity only grows with the logarithm of the number of parameters or the input dimension when the $\ell_{1}$-norm of parameters is well constrained. Then by bounding the norm and the divergence of partial derivatives, we establish that the convergence rate of nonparametric estimation of partial derivatives scales as $\mathcal{O}(n^{-1/4})$, a rate which is slower than the model convergence rate $\mathcal{O}(n^{-1/2})$. To the best of our knowledge, this study combines nonparametric estimation and parametric sparse deep neural networks for the first time. As nonparametric estimation of partial derivatives is of great significance for nonlinear variable selection, the current results show the promising future for the interpretability of deep neural networks.

Nonparametric Estimation via Partial Derivatives

Nonparametric estimation via partial derivatives

Optimal Estimation of Derivatives in Nonparametric Regression

Pathwise Derivatives for Multivariate Distributions

Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression

Estimating Functions and Derivatives Via Adaptive Penalized Splines.

Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimension

On Difference‐based Gradient Estimation in Nonparametric Regression

Parametric-Rate Inference for One-Sided Differentiable Parameters

Sparse and Efficient Estimation for Partial Spline Models with Increasing Dimension

Communication-efficient Distributed Estimation of Partially Linear Additive Models for Large-Scale Data

Semiparametric efficient estimation in high‐dimensional partial linear regression models

Robust Estimation for Partially Linear Models with Large-Dimensional Covariates

An Efficient High-Dimensional Gradient Estimator for Stochastic Differential Equations

Parameter estimation of ODE's via nonparametric estimators

Optimal estimators of cross-partial derivatives and surrogates of functions

Nonparametric Regression Combining Linear Structure

Robust Estimation of Derivatives Using Locally Weighted Least Absolute Deviation Regression

New Methods for Parametric Optimization via Differential Equations

Mean and Covariance Estimation for Discretely Observed High-Dimensional Functional Data: Rates of Convergence and Division of Observational Regimes

High-order empirical interpolation methods for real time solution of parametrized nonlinear PDEs