Abstract:In this article, we propose and develop a novel Bayesian algorithm for optimization of functions whose first and second partial derivatives are known. The basic premise is the Gaussian process representation of the function which induces a first derivative process that is also Gaussian. The Bayesian posterior solutions of the derivative process set equal to zero, given data consisting of suitable choices of input points in the function domain and their function values, emulate the stationary points of the function, which can be fine-tuned by setting restrictions on the prior in terms of the first and second derivatives of the objective function. These observations motivate us to propose a general and effective algorithm for function optimization that attempts to get closer to the true optima adaptively with in-built iterative stages. We provide theoretical foundation to this algorithm, proving almost sure convergence to the true optima as the number of iterative stages tends to infinity. The theoretical foundation hinges upon our proofs of almost sure uniform convergence of the posteriors associated with Gaussian and Gaussian derivative processes to the underlying function and its derivatives in appropriate fixed-domain infill asymptotics setups; rates of convergence are also available. We also provide Bayesian characterization of the number of optima using information inherent in our optimization algorithm. We illustrate our Bayesian optimization algorithm with five different examples involving maxima, minima, saddle points and even inconclusiveness. Our examples range from simple, one-dimensional problems to challenging 50 and 100-dimensional problems.

Local Nonstationarity for Efficient Bayesian Optimization

Bayesian Optimization using Deep Gaussian Processes

The Behavior and Convergence of Local Bayesian Optimization

Simulation Based Bayesian Optimization

Bayesian Optimization for Non-Convex Two-Stage Stochastic Optimization Problems

Pseudo-Bayesian Optimization

Random Exploration in Bayesian Optimization: Order-Optimal Regret and Computational Efficiency

A Study of Bayesian Neural Network Surrogates for Bayesian Optimization

Enhancing Gaussian Process Surrogates for Optimization and Posterior Approximation via Random Exploration

Modulating Surrogates for Bayesian Optimization

BayesOpt: A Library for Bayesian optimization with Robotics Applications

Practical Bayesian Optimization of Machine Learning Algorithms

Recent Advances in Bayesian Optimization

Process Quality Control Through Bayesian Optimization with Adaptive Local Convergence

Function Optimization with Posterior Gaussian Derivative Process

Mastering the exploration-exploitation trade-off in Bayesian Optimization

Bayesian Optimization through Gaussian Cox Process Models for Spatio-temporal Data

Optimistic Optimization of Gaussian Process Samples

Scalable Bayesian Optimization Using Vecchia Approximations of Gaussian Processes

Adaptive Expansion Bayesian Optimization for Unbounded Global Optimization

Information-Theoretic Safe Bayesian Optimization