Abstract:As a non-parametric Bayesian model which produces informative predictive distribution, Gaussian process (GP) has been widely used in various fields, like regression, classification and optimization. The cubic complexity of standard GP however leads to poor scalability, which poses challenges in the era of big data. Hence, various scalable GPs have been developed in the literature in order to improve the scalability while retaining desirable prediction accuracy. This paper devotes to investigating the methodological characteristics and performance of representative global and local scalable GPs including sparse approximations and local aggregations from four main perspectives: scalability, capability, controllability and robustness. The numerical experiments on two toy examples and five real-world datasets with up to 250K points offer the following findings. In terms of scalability, most of the scalable GPs own a time complexity that is linear to the training size. In terms of capability, the sparse approximations capture the long-term spatial correlations, the local aggregations capture the local patterns but suffer from over-fitting in some scenarios. In terms of controllability, we could improve the performance of sparse approximations by simply increasing the inducing size. But this is not the case for local aggregations. In terms of robustness, local aggregations are robust to various initializations of hyperparameters due to the local attention mechanism. Finally, we highlight that the proper hybrid of global and local scalable GPs may be a promising way to improve both the model capability and scalability for big data.

A simple approach for local and global variable importance in nonlinear regression models

Model agnostic local variable importance for locally dependent relationships

A Computational Exploration of Emerging Methods of Variable Importance Estimation

Variable Selection in Logistic Regression Model with Genetic Algorithm.

Reliable and scalable variable importance estimation via warm-start and early stopping

Variable importance in regression models

Variable Selection Using Nonlocal Priors in High-Dimensional Generalized Linear Models With Application to fMRI Data Analysis

Interpreting Deep Neural Networks Through Variable Importance

An additive Gaussian process regression model for interpretable non-parametric analysis of longitudinal data

Enhancing Variable Importance in Random Forests: A Novel Application of Global Sensitivity Analysis

Comparison of feature importance measures as explanations for classification models

Local polynomial regression and variable selection

Lazy Estimation of Variable Importance for Large Neural Networks

The Behavior and Convergence of Local Bayesian Optimization

Model free variable importance for high dimensional data

Variable Importance Assessments and Backward Variable Selection for High-Dimensional Data

Sparsity Oriented Importance Learning for High-dimensional Linear Regression

A Transparent and Nonlinear Method for Variable Selection

Exploring local explanations of nonlinear models using animated linear projections

A screening approach for non-parametric global sensitivity analysis

Understanding and comparing scalable Gaussian process regression for big data