Abstract:We study the problem of estimating the derivatives of a regression function, which has a wide range of applications as a key nonparametric functional of unknown functions. Standard analysis may be tailored to specific derivative orders, and parameter tuning remains a daunting challenge particularly for high-order derivatives. In this article, we propose a simple plug-in kernel ridge regression (KRR) estimator in nonparametric regression with random design that is broadly applicable for multi-dimensional support and arbitrary mixed-partial derivatives. We provide a non-asymptotic analysis to study the behavior of the proposed estimator in a unified manner that encompasses the regression function and its derivatives, leading to two error bounds for a general class of kernels under the strong $L_\infty$ norm. In a concrete example specialized to kernels with polynomially decaying eigenvalues, the proposed estimator recovers the minimax optimal rate up to a logarithmic factor for estimating derivatives of functions in Hölder and Sobolev classes. Interestingly, the proposed estimator achieves the optimal rate of convergence with the same choice of tuning parameter for any order of derivatives. Hence, the proposed estimator enjoys a \textit{plug-in property} for derivatives in that it automatically adapts to the order of derivatives to be estimated, enabling easy tuning in practice. Our simulation studies show favorable finite sample performance of the proposed method relative to several existing methods and corroborate the theoretical findings on its minimax optimality.

K-nearest-neighbour Non-Parametric Estimation of Regression Functions in the Presence of Irrelevant Variables

Local Linear M‐estimation in Non‐parametric Spatial Regression

Improved K nearest neighbor nonparametric regression and its application in short-term traffic flow forecasting

kNN Algorithm for Conditional Mean and Variance Estimation with Automated Uncertainty Quantification and Variable Selection

Automatic and location-adaptive estimation in functional single-index regression

Surprisal Driven $k$-NN for Robust and Interpretable Nonparametric Learning

Efficient Estimation of k for the Nearest Neighbors Class of Methods

The local linear functional kNN estimator of the conditional expectile: uniform consistency in number of neighbors

k-Nearest Neighbors Estimator for Functional Asymmetry Shortfall Regression

Unsupervised K-Nearest Neighbor Regression

Bagging Nearest-Neighbor Prediction Independence Test: an Efficient Method for Nonlinear Dependence of Two Continuous Variables

Distributionally Robust Weighted $k$-Nearest Neighbors

Efficient Estimation for Error Component Seemingly Unrelated Nonparametric Regression Models

Non-uniform Nyström approximation for sparse kernel regression: Theoretical analysis and experimental evaluation

Bayesian Kernel and Mutual $k$-Nearest Neighbor Regression

Spline-Backfitted Kernel Smoothing Of Nonlinear Additive Autoregression Model

Extrapolation Towards Imaginary $0$-Nearest Neighbour and Its Improved Convergence Rate

Improving the Predictive Performances of $k$ Nearest Neighbors Learning by Efficient Variable Selection

Survey of non-parametric estimation methods

On the Estimation of Derivatives Using Plug-in Kernel Ridge Regression Estimators

Efficient Estimation in NPIV Models: A Comparison of Various Neural Networks-Based Estimators