Two Gaussian Approaches to Black-Box Optomization

Lukáš Bajer,Martin Holeňa

DOI: https://doi.org/10.48550/arXiv.1411.7806

2014-11-28

Abstract:Outline of several strategies for using Gaussian processes as surrogate models for the covariance matrix adaptation evolution strategy (CMA-ES).

Neural and Evolutionary Computing,Artificial Intelligence

What problem does this paper attempt to address?

Enhancing Gaussian Process Surrogates for Optimization and Posterior Approximation via Random Exploration

Hwanwoo Kim,Daniel Sanz-Alonso

2024-07-17

Abstract:This paper proposes novel noise-free Bayesian optimization strategies that rely on a random exploration step to enhance the accuracy of Gaussian process surrogate models. The new algorithms retain the ease of implementation of the classical GP-UCB algorithm, but the additional random exploration step accelerates their convergence, nearly achieving the optimal convergence rate. Furthermore, to facilitate Bayesian inference with an intractable likelihood, we propose to utilize optimization iterates for maximum a posteriori estimation to build a Gaussian process surrogate model for the unnormalized log-posterior density. We provide bounds for the Hellinger distance between the true and the approximate posterior distributions in terms of the number of design points. We demonstrate the effectiveness of our Bayesian optimization algorithms in non-convex benchmark objective functions, in a machine learning hyperparameter tuning problem, and in a black-box engineering design problem. The effectiveness of our posterior approximation approach is demonstrated in two Bayesian inference problems for parameters of dynamical systems.

Machine Learning,Numerical Analysis
Global Optimization of Gaussian processes

Artur M. Schweidtmann,Dominik Bongartz,Daniel Grothe,Tim Kerkenhoff,Xiaopeng Lin,Jaromil Najman,Alexander Mitsos

DOI: https://doi.org/10.48550/arXiv.2005.10902

2020-05-21

Optimization and Control

Abstract:Gaussian processes~(Kriging) are interpolating data-driven models that are frequently applied in various disciplines. Often, Gaussian processes are trained on datasets and are subsequently embedded as surrogate models in optimization problems. These optimization problems are nonconvex and global optimization is desired. However, previous literature observed computational burdens limiting deterministic global optimization to Gaussian processes trained on few data points. We propose a reduced-space formulation for deterministic global optimization with trained Gaussian processes embedded. For optimization, the branch-and-bound solver branches only on the degrees of freedom and McCormick relaxations are propagated through explicit Gaussian process models. The approach also leads to significantly smaller and computationally cheaper subproblems for lower and upper bounding. To further accelerate convergence, we derive envelopes of common covariance functions for GPs and tight relaxations of acquisition functions used in Bayesian optimization including expected improvement, probability of improvement, and lower confidence bound. In total, we reduce computational time by orders of magnitude compared to state-of-the-art methods, thus overcoming previous computational burdens. We demonstrate the performance and scaling of the proposed method and apply it to Bayesian optimization with global optimization of the acquisition function and chance-constrained programming. The Gaussian process models, acquisition functions, and training scripts are available open-source within the "MeLOn - Machine Learning Models for Optimization" toolbox~(https://git.rwth-aachen.de/avt.svt/public/MeLOn).
Scalable Bayesian Optimization Using Vecchia Approximations of Gaussian Processes

Felix Jimenez,Matthias Katzfuss

DOI: https://doi.org/10.48550/arXiv.2203.01459

2022-03-03

Abstract:Bayesian optimization is a technique for optimizing black-box target functions. At the core of Bayesian optimization is a surrogate model that predicts the output of the target function at previously unseen inputs to facilitate the selection of promising input values. Gaussian processes (GPs) are commonly used as surrogate models but are known to scale poorly with the number of observations. We adapt the Vecchia approximation, a popular GP approximation from spatial statistics, to enable scalable high-dimensional Bayesian optimization. We develop several improvements and extensions, including training warped GPs using mini-batch gradient descent, approximate neighbor search, and selecting multiple input values in parallel. We focus on the use of our warped Vecchia GP in trust-region Bayesian optimization via Thompson sampling. On several test functions and on two reinforcement-learning problems, our methods compared favorably to the state of the art.

Machine Learning
Bayesian Optimization using Deep Gaussian Processes

Ali Hebbal,Loic Brevault,Mathieu Balesdent,El-Ghazali Talbi,Nouredine Melab

DOI: https://doi.org/10.48550/arXiv.1905.03350

2019-05-07

Abstract:Bayesian Optimization using Gaussian Processes is a popular approach to deal with the optimization of expensive black-box functions. However, because of the a priori on the stationarity of the covariance matrix of classic Gaussian Processes, this method may not be adapted for non-stationary functions involved in the optimization problem. To overcome this issue, a new Bayesian Optimization approach is proposed. It is based on Deep Gaussian Processes as surrogate models instead of classic Gaussian Processes. This modeling technique increases the power of representation to capture the non-stationarity by simply considering a functional composition of stationary Gaussian Processes, providing a multiple layer structure. This paper proposes a new algorithm for Global Optimization by coupling Deep Gaussian Processes and Bayesian Optimization. The specificities of this optimization method are discussed and highlighted with academic test cases. The performance of the proposed algorithm is assessed on analytical test cases and an aerospace design optimization problem and compared to the state-of-the-art stationary and non-stationary Bayesian Optimization approaches.

Machine Learning,Computational Engineering, Finance, and Science
Scaling Gaussian Process Optimization by Evaluating a Few Unique Candidates Multiple Times

Daniele Calandriello,Luigi Carratino,Alessandro Lazaric,Michal Valko,Lorenzo Rosasco

DOI: https://doi.org/10.48550/arXiv.2201.12909

2022-01-31

Abstract:Computing a Gaussian process (GP) posterior has a computational cost cubical in the number of historical points. A reformulation of the same GP posterior highlights that this complexity mainly depends on how many \emph{unique} historical points are considered. This can have important implication in active learning settings, where the set of historical points is constructed sequentially by the learner. We show that sequential black-box optimization based on GPs (GP-Opt) can be made efficient by sticking to a candidate solution for multiple evaluation steps and switch only when necessary. Limiting the number of switches also limits the number of unique points in the history of the GP. Thus, the efficient GP reformulation can be used to exactly and cheaply compute the posteriors required to run the GP-Opt algorithms. This approach is especially useful in real-world applications of GP-Opt with high switch costs (e.g. switching chemicals in wet labs, data/model loading in hyperparameter optimization). As examples of this meta-approach, we modify two well-established GP-Opt algorithms, GP-UCB and GP-EI, to switch candidates as infrequently as possible adapting rules from batched GP-Opt. These versions preserve all the theoretical no-regret guarantees while improving practical aspects of the algorithms such as runtime, memory complexity, and the ability of batching candidates and evaluating them in parallel.

Machine Learning
An Adaptive Re-evaluation Method for Evolution Strategy under Additive Noise

Catalin-Viorel Dinu,Yash J. Patel,Xavier Bonet-Monroig,Hao Wang

2024-09-25

Abstract:The Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) is one of the most advanced algorithms in numerical black-box optimization. For noisy objective functions, several approaches were proposed to mitigate the noise, e.g., re-evaluations of the same solution or adapting the population size. In this paper, we devise a novel method to adaptively choose the optimal re-evaluation number for function values corrupted by additive Gaussian white noise. We derive a theoretical lower bound of the expected improvement achieved in one iteration of CMA-ES, given an estimation of the noise level and the Lipschitz constant of the function's gradient. Solving for the maximum of the lower bound, we obtain a simple expression of the optimal re-evaluation number. We experimentally compare our method to the state-of-the-art noise-handling methods for CMA-ES on a set of artificial test functions across various noise levels, optimization budgets, and dimensionality. Our method demonstrates significant advantages in terms of the probability of hitting near-optimal function values.

Neural and Evolutionary Computing
A Solution to the Ill-Conditioning of Gradient-Enhanced Covariance Matrices for Gaussian Processes

André L. Marchildon,David W. Zingg

2023-07-12

Abstract:Gaussian processes provide probabilistic surrogates for various applications including classification, uncertainty quantification, and optimization. Using a gradient-enhanced covariance matrix can be beneficial since it provides a more accurate surrogate relative to its gradient-free counterpart. An acute problem for Gaussian processes, particularly those that use gradients, is the ill-conditioning of their covariance matrices. Several methods have been developed to address this problem for gradient-enhanced Gaussian processes but they have various drawbacks such as limiting the data that can be used, imposing a minimum distance between evaluation points in the parameter space, or constraining the hyperparameters. In this paper a new method is presented that applies a diagonal preconditioner to the covariance matrix along with a modest nugget to ensure that the condition number of the covariance matrix is bounded, while avoiding the drawbacks listed above. Optimization results for a gradient-enhanced Bayesian optimizer with the Gaussian kernel are compared with the use of the new method, a baseline method that constrains the hyperparameters, and a rescaling method that increases the distance between evaluation points. The Bayesian optimizer with the new method converges the optimality, ie the $\ell_2$ norm of the gradient, an additional 5 to 9 orders of magnitude relative to when the baseline method is used and it does so in fewer iterations than with the rescaling method. The new method is available in the open source python library GpGradPy, which can be found at <a class="link-external link-https" href="https://github.com/marchildon/gpgradpy/tree/paper_precon" rel="external noopener nofollow">this https URL</a>. All of the figures in this paper can be reproduced with this library.

Optimization and Control
Generative Evolutionary Strategy For Black-Box Optimizations

Changhwi Park

2024-01-27

Abstract:Many scientific and technological problems are related to optimization. Among them, black-box optimization in high-dimensional space is particularly challenging. Recent neural network-based black-box optimization studies have shown noteworthy achievements. However, their capability in high-dimensional search space is still limited. This study proposes a black-box optimization method based on the evolution strategy (ES) and the generative neural network (GNN) model. We designed the algorithm so that the ES and the GNN model work cooperatively. This hybrid model enables reliable training of surrogate networks; it optimizes multi-objective, high-dimensional, and stochastic black-box functions. Our method outperforms baseline optimization methods in this experiment, including ES, and Bayesian optimization.

Neural and Evolutionary Computing
Pseudo-Bayesian Optimization

Haoxian Chen,Henry Lam

2024-06-21

Abstract:Bayesian Optimization is a popular approach for optimizing expensive black-box functions. Its key idea is to use a surrogate model to approximate the objective and, importantly, quantify the associated uncertainty that allows a sequential search of query points that balance exploitation-exploration. Gaussian process (GP) has been a primary candidate for the surrogate model, thanks to its Bayesian-principled uncertainty quantification power and modeling flexibility. However, its challenges have also spurred an array of alternatives whose convergence properties could be more opaque. Motivated by these, we study in this paper an axiomatic framework that elicits the minimal requirements to guarantee black-box optimization convergence that could apply beyond GP-based methods. Moreover, we leverage the design freedom in our framework, which we call Pseudo-Bayesian Optimization, to construct empirically superior algorithms. In particular, we show how using simple local regression, and a suitable "randomized prior" construction to quantify uncertainty, not only guarantees convergence but also consistently outperforms state-of-the-art benchmarks in examples ranging from high-dimensional synthetic experiments to realistic hyperparameter tuning and robotic applications.

Machine Learning
Optimistic Optimization of Gaussian Process Samples

Julia Grosse,Cheng Zhang,Philipp Hennig

DOI: https://doi.org/10.48550/arXiv.2209.00895

IF: 5.414

2022-09-02

Machine Learning

Abstract:Bayesian optimization is a popular formalism for global optimization, but its computational costs limit it to expensive-to-evaluate functions. A competing, computationally more efficient, global optimization framework is optimistic optimization, which exploits prior knowledge about the geometry of the search space in form of a dissimilarity function. We investigate to which degree the conceptual advantages of Bayesian Optimization can be combined with the computational efficiency of optimistic optimization. By mapping the kernel to a dissimilarity, we obtain an optimistic optimization algorithm for the Bayesian Optimization setting with a run-time of up to $\mathcal{O}(N \log N)$. As a high-level take-away we find that, when using stationary kernels on objectives of relatively low evaluation cost, optimistic optimization can be strongly preferable over Bayesian optimization, while for strongly coupled and parametric models, good implementations of Bayesian optimization can perform much better, even at low evaluation cost. We argue that there is a new research domain between geometric and probabilistic search, i.e. methods that run drastically faster than traditional Bayesian optimization, while retaining some of the crucial functionality of Bayesian optimization.
Gaussian process optimization with failures: classification and convergence proof

François Bachoc,Céline Helbert,Victor Picheny

DOI: https://doi.org/10.1007/s10898-020-00920-0

2020-07-08

Journal of Global Optimization

Abstract:We consider the optimization of a computer model where each simulation either fails or returns a valid output performance. We first propose a new joint Gaussian process model for classification of the inputs (computation failure or success) and for regression of the performance function. We provide results that allow for a computationally efficient maximum likelihood estimation of the covariance parameters, with a stochastic approximation of the likelihood gradient. We then extend the classical improvement criterion to our setting of joint classification and regression. We provide an efficient computation procedure for the extended criterion and its gradient. We prove the almost sure convergence of the global optimization algorithm following from this extended criterion. We also study the practical performances of this algorithm, both on simulated data and on a real computer model in the context of automotive fan design.

mathematics, applied,operations research & management science
Landscape Analysis for Surrogate Models in the Evolutionary Black-Box Context

Zbyněk Pitra,Jan Koza,Jiří Tumpach,Martin Holeňa

DOI: https://doi.org/10.48550/arXiv.2203.11315

2022-10-03

Abstract:Surrogate modeling has become a valuable technique for black-box optimization tasks with expensive evaluation of the objective function. In this paper, we investigate the relationship between the predictive accuracy of surrogate models and features of the black-box function landscape. We also study properties of features for landscape analysis in the context of different transformations and ways of selecting the input data. We perform the landscape analysis of a large set of data generated using runs of a surrogate-assisted version of the Covariance Matrix Adaptation Evolution Strategy on the noiseless part of the Comparing Continuous Optimisers benchmark function testbed.

Neural and Evolutionary Computing,Machine Learning,Optimization and Control
Optimization of Gaussian Random Fields

Eric Dow,Qiqi Wang

DOI: https://doi.org/10.48550/arXiv.1407.1857

2014-07-08

Abstract:Many engineering systems are subject to spatially distributed uncertainty, i.e. uncertainty that can be modeled as a random field. Altering the mean or covariance of this uncertainty will in general change the statistical distribution of the system outputs. We present an approach for computing the sensitivity of the statistics of system outputs with respect to the parameters describing the mean and covariance of the distributed uncertainty. This sensitivity information is then incorporated into a gradient-based optimizer to optimize the structure of the distributed uncertainty to achieve desired output statistics. This framework is applied to perform variance optimization for a model problem and to optimize the manufacturing tolerances of a gas turbine compressor blade.

Optimization and Control
Adaptive Gradient Enhanced Gaussian Process Surrogates for Inverse Problems

Phillip Semler,Martin Weiser

2024-04-02

Abstract:Generating simulated training data needed for constructing sufficiently accurate surrogate models to be used for efficient optimization or parameter identification can incur a huge computational effort in the offline phase. We consider a fully adaptive greedy approach to the computational design of experiments problem using gradient-enhanced Gaussian process regression as surrogates. Designs are incrementally defined by solving an optimization problem for accuracy given a certain computational budget. We address not only the choice of evaluation points but also of required simulation accuracy, both of values and gradients of the forward model. Numerical results show a significant reduction of the computational effort compared to just position-adaptive and static designs as well as a clear benefit of including gradient information into the surrogate training.

Numerical Analysis
Scalable Gaussian process-based transfer surrogates for hyperparameter optimization

Martin Wistuba,Nicolas Schilling,Lars Schmidt-Thieme

DOI: https://doi.org/10.1007/s10994-017-5684-y

IF: 5.414

2017-12-22

Machine Learning

Abstract:Algorithm selection as well as hyperparameter optimization are tedious task that have to be dealt with when applying machine learning to real-world problems. Sequential model-based optimization (SMBO), based on so-called “surrogate models”, has been employed to allow for faster and more direct hyperparameter optimization. A surrogate model is a machine learning regression model which is trained on the meta-level instances in order to predict the performance of an algorithm on a specific data set given the hyperparameter settings and data set descriptors. Gaussian processes, for example, make good surrogate models as they provide probability distributions over labels. Recent work on SMBO also includes meta-data, i.e. observed hyperparameter performances on other data sets, into the process of hyperparameter optimization. This can, for example, be accomplished by learning transfer surrogate models on all available instances of meta-knowledge; however, the increasing amount of meta-information can make Gaussian processes infeasible, as they require the inversion of a large covariance matrix which grows with the number of instances. Consequently, instead of learning a joint surrogate model on all of the meta-data, we propose to learn individual surrogate models on the observations of each data set and then combine all surrogates to a joint one using ensembling techniques. The final surrogate is a weighted sum of all data set specific surrogates plus an additional surrogate that is solely learned on the target observations. Within our framework, any surrogate model can be used and explore Gaussian processes in this scenario. We present two different strategies for finding the weights used in the ensemble: the first is based on a probabilistic product of experts approach, and the second is based on kernel regression. Additionally, we extend the framework to directly estimate the acquisition function in the same setting, using a novel technique which we name the “transfer acquisition function”. In an empirical evaluation including comparisons to the current state-of-the-art on two publicly available meta-data sets, we are able to demonstrate that our proposed approach does not only scale to large meta-data, but also finds the stronger prediction models.

computer science, artificial intelligence
Parallel Gaussian Process Optimization with Upper Confidence Bound and Pure Exploration

Emile Contal,David Buffoni,Alexandre Robicquet,Nicolas Vayatis

DOI: https://doi.org/10.1007/978-3-642-40988-2_15

2013-09-02

Abstract:In this paper, we consider the challenge of maximizing an unknown function f for which evaluations are noisy and are acquired with high cost. An iterative procedure uses the previous measures to actively select the next estimation of f which is predicted to be the most useful. We focus on the case where the function can be evaluated in parallel with batches of fixed size and analyze the benefit compared to the purely sequential procedure in terms of cumulative regret. We introduce the Gaussian Process Upper Confidence Bound and Pure Exploration algorithm (GP-UCB-PE) which combines the UCB strategy and Pure Exploration in the same batch of evaluations along the parallel iterations. We prove theoretical upper bounds on the regret with batches of size K for this procedure which show the improvement of the order of sqrt{K} for fixed iteration cost over purely sequential versions. Moreover, the multiplicative constants involved have the property of being dimension-free. We also confirm empirically the efficiency of GP-UCB-PE on real and synthetic problems compared to state-of-the-art competitors.

Machine Learning
Noisy Multiobjective Black-Box Optimization Using Bayesian Optimization.

Hongyan Wang,Hua Xu,Yuan,Junhui Deng,Xiaomin Sun

DOI: https://doi.org/10.1145/3319619.3321898

2019-01-01

Abstract:Expensive black-box problems are usually optimized by Bayesian Optimization (BO) since it can reduce evaluation costs via cheaper surrogates. The most popular model used in Bayesian Optimization is the Gaussian process (GP) whose posterior is based on a joint GP prior built by initial observations, so the posterior is also a Gaussian process. Observations are often not noise-free, so in most of these cases, a noisy transformation of the objective space is observed. Many single objective optimization algorithms have succeeded in extending efficient global optimization (EGO) to noisy circumstances, while ParEGO fails to consider noise. In order to deal with noisy expensive black-box problems, we extending ParEGO to noisy optimization according to adding a Gaussian noisy error while approximating the surrogate. We call it noisy-ParEGO and results of S -metric indicate that the algorithm works well on optimizing noisy expensive multiobjective black-box problems.
High-dimensional Bayesian Optimization via Covariance Matrix Adaptation Strategy

Lam Ngo,Huong Ha,Jeffrey Chan,Vu Nguyen,Hongyu Zhang

2024-02-05

Abstract:Bayesian Optimization (BO) is an effective method for finding the global optimum of expensive black-box functions. However, it is well known that applying BO to high-dimensional optimization problems is challenging. To address this issue, a promising solution is to use a local search strategy that partitions the search domain into local regions with high likelihood of containing the global optimum, and then use BO to optimize the objective function within these regions. In this paper, we propose a novel technique for defining the local regions using the Covariance Matrix Adaptation (CMA) strategy. Specifically, we use CMA to learn a search distribution that can estimate the probabilities of data points being the global optimum of the objective function. Based on this search distribution, we then define the local regions consisting of data points with high probabilities of being the global optimum. Our approach serves as a meta-algorithm as it can incorporate existing black-box BO optimizers, such as BO, TuRBO, and BAxUS, to find the global optimum of the objective function within our derived local regions. We evaluate our proposed method on various benchmark synthetic and real-world problems. The results demonstrate that our method outperforms existing state-of-the-art techniques.

Machine Learning
Mastering the exploration-exploitation trade-off in Bayesian Optimization

Antonio Candelieri

2023-05-15

Abstract:Gaussian Process based Bayesian Optimization is a well-known sample efficient sequential strategy for globally optimizing black-box, expensive, and multi-extremal functions. The role of the Gaussian Process is to provide a probabilistic approximation of the unknown function, depending on the sequentially collected observations, while an acquisition function drives the choice of the next solution to evaluate, balancing between exploration and exploitation, depending on the current Gaussian Process model. Despite the huge effort of the scientific community in defining effective exploration-exploitation mechanisms, we are still far away from the master acquisition function. This paper merges the most relevant results and insights from both algorithmic and human search strategies to propose a novel acquisition function, mastering the trade-off between explorative and exploitative choices, adaptively. We compare the proposed acquisition function on a number of test functions and against different state-of-the-art ones, which are instead based on prefixed or random scheduling between exploration and exploitation. A Pareto analysis is performed with respect to two (antagonistic) goals: convergence to the optimum and exploration capability. Results empirically prove that the proposed acquisition function is almost always Pareto optimal and also the most balanced trade-off between the two goals.

Machine Learning,Artificial Intelligence,Optimization and Control
Vector Optimization with Gaussian Process Bandits

İlter Onat Korkmaz,Yaşar Cahit Yıldırım,Çağın Ararat,Cem Tekin

2024-12-03

Abstract:Learning problems in which multiple conflicting objectives must be considered simultaneously often arise in various fields, including engineering, drug design, and environmental management. Traditional methods for dealing with multiple black-box objective functions, such as scalarization and identification of the Pareto set under the componentwise order, have limitations in incorporating objective preferences and exploring the solution space accordingly. While vector optimization offers improved flexibility and adaptability via specifying partial orders based on ordering cones, current techniques designed for sequential experiments either suffer from high sample complexity or lack theoretical guarantees. To address these issues, we propose Vector Optimization with Gaussian Process (VOGP), a probably approximately correct adaptive elimination algorithm that performs black-box vector optimization using Gaussian process bandits. VOGP allows users to convey objective preferences through ordering cones while performing efficient sampling by exploiting the smoothness of the objective function, resulting in a more effective optimization process that requires fewer evaluations. We establish theoretical guarantees for VOGP and derive information gain-based and kernel-specific sample complexity bounds. We also conduct experiments on both real-world and synthetic datasets to compare VOGP with the state-of-the-art methods.

Machine Learning,Applications

Two Gaussian Approaches to Black-Box Optomization

Enhancing Gaussian Process Surrogates for Optimization and Posterior Approximation via Random Exploration

Global Optimization of Gaussian processes

Scalable Bayesian Optimization Using Vecchia Approximations of Gaussian Processes

Bayesian Optimization using Deep Gaussian Processes

Scaling Gaussian Process Optimization by Evaluating a Few Unique Candidates Multiple Times

An Adaptive Re-evaluation Method for Evolution Strategy under Additive Noise

A Solution to the Ill-Conditioning of Gradient-Enhanced Covariance Matrices for Gaussian Processes

Generative Evolutionary Strategy For Black-Box Optimizations

Pseudo-Bayesian Optimization

Optimistic Optimization of Gaussian Process Samples

Gaussian process optimization with failures: classification and convergence proof

Landscape Analysis for Surrogate Models in the Evolutionary Black-Box Context

Optimization of Gaussian Random Fields

Adaptive Gradient Enhanced Gaussian Process Surrogates for Inverse Problems

Scalable Gaussian process-based transfer surrogates for hyperparameter optimization

Parallel Gaussian Process Optimization with Upper Confidence Bound and Pure Exploration

Noisy Multiobjective Black-Box Optimization Using Bayesian Optimization.

High-dimensional Bayesian Optimization via Covariance Matrix Adaptation Strategy

Mastering the exploration-exploitation trade-off in Bayesian Optimization

Vector Optimization with Gaussian Process Bandits