Expected Coordinate Improvement for High-Dimensional Bayesian Optimization

Dawei Zhan

2024-04-18

Abstract:Bayesian optimization (BO) algorithm is very popular for solving low-dimensional expensive optimization problems. Extending Bayesian optimization to high dimension is a meaningful but challenging task. One of the major challenges is that it is difficult to find good infill solutions as the acquisition functions are also high-dimensional. In this work, we propose the expected coordinate improvement (ECI) criterion for high-dimensional Bayesian optimization. The proposed ECI criterion measures the potential improvement we can get by moving the current best solution along one coordinate. The proposed approach selects the coordinate with the highest ECI value to refine in each iteration and covers all the coordinates gradually by iterating over the coordinates. The greatest advantage of the proposed ECI-BO (expected coordinate improvement based Bayesian optimization) algorithm over the standard BO algorithm is that the infill selection problem of the proposed algorithm is always a one-dimensional problem thus can be easily solved. Numerical experiments show that the proposed algorithm can achieve significantly better results than the standard BO algorithm and competitive results when compared with five state-of-the-art high-dimensional BOs. This work provides a simple but efficient approach for high-dimensional Bayesian optimization.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

The paper focuses on the problem of high-dimensional Bayesian Optimization (BO), which is a challenging task widely applied in low dimensions. The main challenge lies in finding good infill solutions as the dimension increases, because the acquisition function is also high-dimensional. The paper proposes a new criterion called Expected Coordinate Improvement (ECI), which measures the improvement that can be obtained by moving along a coordinate axis from the current best solution. The algorithm optimizes by selecting the coordinate with the highest ECI value at each iteration, gradually covering all coordinates. Compared to standard BO algorithms, the advantage of the ECI-BO algorithm is that its infill selection is always a one-dimensional problem, making it easier to solve.

The paper mentions various methods for solving high-dimensional optimization problems, including variable selection, embedding, and decomposition strategies. Although some methods such as random embedding, decomposition, or linear dimension reduction exist, they often assume specific structures of the objective function, which may not hold in practical applications. The ECI-BO algorithm provides a simple and effective approach by gradually solving high-dimensional problems through optimizing one coordinate at a time, resulting in a different search trajectory from standard BO.

Numerical experiments show that the ECI-BO algorithm exhibits significantly better results or competitiveness compared to standard BO algorithms and five state-of-the-art high-dimensional BO algorithms. The authors also note that the ECI-BO algorithm introduces a small computational cost but significantly improves optimization performance.

The main contribution of the paper is the proposal of the ECI criterion and the ECI-BO algorithm based on this criterion, offering a new approach to solve high-dimensional expensive optimization problems.