Abstract:Optimization of problems with high computational power demands is a challenging task. A probabilistic approach to such optimization called Bayesian optimization lowers performance demands by solving mathematically simpler model of the problem. Selected approach, Gaussian Process, models problem using a mixture of Gaussian functions. This paper presents specific modifications of Gaussian Process optimization components from available scientific libraries. Presented modifications were submitted to BlackBox 2020 challenge, where it outperformed some conventionally available optimization libraries.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the efficient solution of high - computational - demand optimization problems. Specifically, the paper focuses on how to reduce the computational cost and time consumption in the optimization process through the Bayesian Optimization method, especially when dealing with large - scale data and complex models. ### Problem Background With the wide application of machine learning (ML) methods in multiple research fields, choosing the best ML method and tuning its parameters (i.e., hyper - parameter optimization or "hyper - optimization") has become a common way to achieve state - of - the - art results. However, as the amount of data increases and the complexity of models rises, hyper - parameter optimization will quickly exhaust the available computational resources. For example, systematically optimizing deep - learning (DL) training parameters may take several days. ### Solution To solve this problem, the paper proposes a Bayesian optimization method based on the Gaussian Process (GP) and makes specific improvements to it. Bayesian optimization reduces the number of required samples by constructing a surrogate model of the objective function, thereby reducing the computational cost. As a probability model, GP can approximate the objective function by mixing Gaussian functions and is optimized in combination with the Differential Evolution Algorithm. ### Main Contributions 1. **Improvement of Gaussian Process Optimization Components**: The paper makes specific modifications to the existing GP optimization components in scientific libraries and applies them to the BlackBox 2020 challenge, achieving results superior to some traditional optimization libraries. 2. **Handling of Non - continuous Parameters**: The GP is extended to handle non - continuous parameters (such as categories, integers, and Boolean values), ensuring that these parameters are not extrapolated to undefined values. 3. **Parallelized Optimization**: The parallelization of GP is achieved by introducing the kriging believer method, which improves the optimization efficiency. 4. **Initialization Strategy**: Advanced initialization methods such as Latin Hypercube Sampling (LH) are adopted to improve the stability and performance of optimization. ### Summary The main purpose of this paper is to provide a more efficient optimization framework by improving the Gaussian process method in Bayesian optimization, which is especially suitable for tasks with high computational requirements. These improvements not only reduce the computational cost but also demonstrate good performance in the BlackBox 2020 challenge and finally achieve an 11th - place finish.

Hyper-optimization with Gaussian Process and Differential Evolution Algorithm

A Method of Adaptive Hyperparameter Optimization for Deep Generative Models

Two Gaussian Approaches to Black-Box Optomization

Optimal Designs of Gaussian Processes with Budgets for Hyperparameter Optimization

Batched Gaussian Process Bandit Optimization via Determinantal Point Processes

Provably Efficient Bayesian Optimization with Unbiased Gaussian Process Hyperparameter Estimation

Optimistic Optimization of Gaussian Process Samples

Global Optimization of Gaussian processes

Gaussian Processes and Bayesian Optimization for High Precision Experiments

Practical Bayesian Optimization of Machine Learning Algorithms

Scalable Bayesian Optimization Using Vecchia Approximations of Gaussian Processes

Bayesian Optimization with High-Dimensional Outputs

Bayesian Optimization using Deep Gaussian Processes

Function Optimization with Posterior Gaussian Derivative Process

Process Quality Control Through Bayesian Optimization with Adaptive Local Convergence

Pseudo-Bayesian Optimization

Deep Gaussian process for multi-objective Bayesian optimization

Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

Enhancing Optimization Performance: A Novel Hybridization of Gaussian Crunching Search and Powell's Method for Derivative-Free Optimization

A Solution to the Ill-Conditioning of Gradient-Enhanced Covariance Matrices for Gaussian Processes