Abstract:Kernel methods are attractive in data analysis as they can model nonlinear similarities between observations and provide means to rich representations, both of which are useful for the regression problems in general domains. Despite their popularity, they suffer from two primary inherent drawbacks. One drawback is the positive definiteness requirement of the kernel functions, which greatly restricts their applications to some real data analysis. The other drawback is their poor scalability in massive data scenarios. In this paper, we aim to address these two problems by considering the Nystrom subsampling approach for coefficient-based regularized regression (or Nystrom CRR for short). Nystrom subsampling is an effective approach to reduce the space and time complexity by constructing a low-rank approximation of the original kernel matrix through column subsampling. Coefficient-based regularized regression can provide a simple paradigm for designing indefinite kernel methods. We show that a combination of these two schemes is not only computationally efficient but also statistically consistent with a mini-max optimal rates of convergence. We explicitly determine the lower bound of subsampling level as a function of the sample size such that the mini-max optimal convergence rates can be preserved. From our analysis, the subsampling level plays a role as a trade-off between computational and asymptotic behaviors of Nystrom CRR, and hence is pivotal for algorithmic performances. In order to choose an appropriate subsampling level, we develop an incremental Nystrom CRR for nested subsampling sets. The proposed algorithm can greatly reduce the cost of cross-validation, and even allows to compute the whole solution path through all possible subsampling levels. Based on our empirical studies, the incremental Nystrom CRR can perform effective model selections and achieve the state-of-the-art results on both synthetic and real data sets.

Analysis of Regularized Nyström Subsampling for Regression Functions of Low Smoothness

Nyström Subsampling Method for Coefficient-Based Regularized Regression

Convergence Analysis of regularised Nyström method for Functional Linear Regression

Non-uniform Nyström approximation for sparse kernel regression: Theoretical analysis and experimental evaluation

Reduced Heteroscedasticity Linear Regression for Nyström Approximation

Analysis of Least Squares Regularized Regression in Reproducing Kernel Krein Spaces

Analysis of Regularized Least-Squares in Reproducing Kernel Kreĭn Spaces

An Analysis of Tikhonov Regularization for Nonlinear Ill-Posed Problems under a General Smoothness Assumption

Nyström Regularization for Time Series Forecasting

Manifold regularization based on Nystr{ö}m type subsampling

Learning Rates of Least-Square Regularized Regression with Polynomial Kernels

Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression

Learning Theory Estimates for Coefficient-Based Regularized Regression

Error analysis of regularized trigonometric linear regression with unbounded sampling: a statistical learning viewpoint

Small noise analysis for Tikhonov and RKHS regularizations

Random Smoothing Regularization in Kernel Gradient Descent Learning

Improved Bounds for the Nyström Method with Application to Kernel Classification

Sparse Kernel Regression with Coefficient-based $\Ell_q-$Regularization.

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

On regularized Radon-Nikodym differentiation