Abstract:We study the problem of estimating the derivatives of a regression function, which has a wide range of applications as a key nonparametric functional of unknown functions. Standard analysis may be tailored to specific derivative orders, and parameter tuning remains a daunting challenge particularly for high-order derivatives. In this article, we propose a simple plug-in kernel ridge regression (KRR) estimator in nonparametric regression with random design that is broadly applicable for multi-dimensional support and arbitrary mixed-partial derivatives. We provide a non-asymptotic analysis to study the behavior of the proposed estimator in a unified manner that encompasses the regression function and its derivatives, leading to two error bounds for a general class of kernels under the strong $L_\infty$ norm. In a concrete example specialized to kernels with polynomially decaying eigenvalues, the proposed estimator recovers the minimax optimal rate up to a logarithmic factor for estimating derivatives of functions in Hölder and Sobolev classes. Interestingly, the proposed estimator achieves the optimal rate of convergence with the same choice of tuning parameter for any order of derivatives. Hence, the proposed estimator enjoys a \textit{plug-in property} for derivatives in that it automatically adapts to the order of derivatives to be estimated, enabling easy tuning in practice. Our simulation studies show favorable finite sample performance of the proposed method relative to several existing methods and corroborate the theoretical findings on its minimax optimality.

Quick and Simple Kernel Differential Equation Regression Estimators for Data with Sparse Design

Discovery of differential equations using sparse state and parameter regression

Kernel Ordinary Differential Equations

Numerical Discretization-Based Kernel Type Estimation Methods for Ordinary Differential Equation Models

Local Linear Smoothers Using Asymmetric Kernels

A Learning Approach to Derive Sparse Kernel Minimum Square Error Model

Learning Differential Equations that are Easy to Solve

Nonparametric estimation of the log odds ratio for sparse data by kernel smoothing

Stahel-Donoho Kernel Estimation for Fixed Design Nonparametric Regression Models

Sparse Regression for Low-Dimensional Time-Dynamic Varying Coefficient Models with Application to Air Quality Data

Non-uniform Nyström approximation for sparse kernel regression: Theoretical analysis and experimental evaluation

On the Estimation of Derivatives Using Plug-in Kernel Ridge Regression Estimators

Parameter Estimation for Differential Equations: a Generalized Smoothing Approach

Sparse Nonlinear Regression: Parameter Estimation and Asymptotic Inference

Optimal Estimation of Derivatives in Nonparametric Regression

Automatically discovering ordinary differential equations from data with sparse regression

Parameter Estimation of Linear Stochastic Differential Equations with Sparse Observations

Kernel Regression-Based State-Space Estimation of Continuous-Time Dynamic Systems From Noisy Sampled State Data

Dynamical Modeling for Non-Gaussian Data with High-Dimensional Sparse Ordinary Differential Equations

LOCALLY SPARSE ESTIMATOR OFGENERALIZED VARYING COEFFICIENT MODELFOR ASYNCHRONOUS LONGITUDINAL DATA

Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed!