Abstract:Despite the widespread utilization of Gaussian process models for versatile nonparametric modeling, they exhibit limitations in effectively capturing abrupt changes in function smoothness and accommodating relationships with heteroscedastic errors. Addressing these shortcomings, the heteroscedastic Gaussian process (HeGP) regression seeks to introduce flexibility by acknowledging the variability of residual variances across covariates in the regression model. In this work, we extend the HeGP concept, expanding its scope beyond regression tasks to encompass classification and state-space models. To achieve this, we propose a novel framework where the Gaussian process is coupled with a covariate-induced precision matrix process, adopting a mixture formulation. This approach enables the modeling of heteroscedastic covariance functions across covariates. To mitigate the computational challenges posed by sampling, we employ variational inference to approximate the posterior and facilitate posterior predictive modeling. Additionally, our training process leverages an EM algorithm featuring closed-form M-step updates to efficiently evaluate the heteroscedastic covariance function. A notable feature of our model is its consistent performance on multivariate responses, accommodating various types (continuous or categorical) seamlessly. Through a combination of simulations and real-world applications in climatology, we illustrate the model's prowess and advantages. By overcoming the limitations of traditional Gaussian process models, our proposed framework offers a robust and versatile tool for a wide array of applications.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to address the limitations of traditional Gaussian Process Models (GPs) in handling abrupt changes in function smoothness and heteroscedastic error relationships. Specifically: 1. **Abrupt Changes in Function Smoothness**: Traditional GP models perform poorly in capturing abrupt changes in function smoothness, which limits their flexibility and accuracy in certain application scenarios. 2. **Heteroscedastic Errors**: Traditional GP models assume constant error variance and cannot effectively handle situations where error variance changes with covariates, i.e., heteroscedasticity. To solve these issues, the paper introduces the Heteroscedastic Gaussian Process (HeGP) regression model. This model extends the capabilities of traditional GP models by incorporating a covariate-induced precision matrix process, enabling it to handle heteroscedastic errors. Additionally, the paper further extends the application scope of the HeGP model to include classification and state-space models. ### Main Contributions 1. **Model Extension**: A novel framework is proposed that combines Gaussian Processes with a covariate-induced precision matrix process in a hybrid form, allowing the model to handle multi-response data (including both continuous and categorical types). 2. **Computational Methods**: To address the computational challenges posed by sampling, a variational inference method is employed to approximate the posterior distribution and facilitate posterior predictive modeling. During training, the EM algorithm is utilized, with closed-form M-step updates for efficient evaluation of the heteroscedastic covariance function. 3. **Performance Validation**: The advantages and strong performance of the model are demonstrated through simulations and real-world applications (such as climatology), proving its robustness and flexibility in various applications. ### Key Techniques - **Heteroscedastic Gaussian Process**: Handles heteroscedastic errors by incorporating a covariate-induced precision matrix process. - **Variational Inference**: Used to approximate the posterior distribution, improving computational efficiency. - **EM Algorithm**: Used for parameter estimation, with closed-form M-step updates to enhance computational speed and stability. In summary, this paper overcomes the limitations of traditional GP models by proposing a new heteroscedastic Gaussian process model and its inference methods, providing a powerful tool for non-parametric heteroscedastic modeling.

Multi-Response Heteroscedastic Gaussian Process Models and Their Inference

Residual Gaussian Process: A Tractable Nonparametric Bayesian Emulator for Multi-Fidelity Simulations

Large-Scale Heteroscedastic Regression via Gaussian Process

Infinite-Horizon Gaussian Processes

Towards Efficient Modeling and Inference in Multi-Dimensional Gaussian Process State-Space Models

Prediction Using a Bayesian Heteroscedastic Composite Gaussian Process

Scalable mixed-domain Gaussian process modeling and model reduction for longitudinal data

Heterogeneous Multi-Task Gaussian Cox Processes.

A comprehensive multi-fidelity surrogate framework based on Gaussian process for datasets with heterogeneous responses

Learning Multi-Task Gaussian Process over Heterogeneous Input Domains

Multivariate Gaussian and Student$-t$ Process Regression for Multi-output Prediction

Data-Driven Design via Scalable Gaussian Processes for Multi-Response Big Data With Qualitative Factors

Bayesian Learning of Orthogonal Embeddings for Multi-Fidelity Gaussian Processes

Latent Gaussian Process Regression

Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification

Nonparametric Bayesian Mixed-effect Model: a Sparse Gaussian Process Approach

Dimensionality Reduction for Multi-Fidelity Gaussian Processes Using Bayesian Adaptation

Towards Scalable Gaussian Process Modeling

Mixed-Stationary Gaussian Process for Flexible Non-Stationary Modeling of Spatial Outcomes

Gaussian predictive process models for large spatial data sets

Simultaneous Modeling of Multivariate Heterogeneous Responses and Heteroskedasticity Via a Two‐stage Composite Likelihood