Abstract:Heuristic optimisation algorithms explore the search space by sampling solutions, evaluating their fitness, and biasing the search in the direction of promising solutions. However, in many cases, this fitness function involves executing expensive computational calculations, drastically reducing the reasonable number of evaluations. In this context, surrogate models have emerged as an excellent alternative to alleviate these computational problems. This paper addresses the formulation of surrogate problems as both regression models that approximate fitness (surface surrogate models) and a novel way to connect classification models (pairwise surrogate models). The pairwise approach can be directly exploited by some algorithms, such as Differential Evolution, in which the fitness value is not actually needed to drive the search, and it is sufficient to know whether a solution is better than another one or not. Based on these modelling approaches, we have conducted a multidimensional analysis of surrogate models under different configurations: different machine learning algorithms (regularised regression, neural networks, decision trees, boosting methods, and random forests), different surrogate strategies (encouraging diversity or relaxing prediction thresholds), and compare them for both surface and pairwise surrogate models. The experimental part of the article includes the benchmark problems already proposed for the SOCO2011 competition in continuous optimisation and a simulation problem included in the recent GECCO2021 Industrial Challenge. This paper shows that the performance of the overall search, when using online machine learning-based surrogate models, depends not only on the accuracy of the predictive model but also on both the kind of bias towards positive or negative cases and how the optimisation uses those predictions to decide whether to execute the actual fitness function.

CODES: Benchmarking Coupled ODE Surrogates

Composite Differential Evolution with Modified Oracle Penalty Method for Constrained Optimization Problems

COCO: a platform for comparing continuous optimizers in a black-box setting

Codabench: Flexible, Easy-to-use, and Reproducible Meta-Benchmark Platform

Programming with Neural Surrogates of Programs

DiCE: The Infinitely Differentiable Monte-Carlo Estimator

CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks

SurCo: Learning Linear Surrogates For Combinatorial Nonlinear Optimization Problems

Adaptive Multi-Objective Differential Evolution with Stochastic Coding Strategy.

Comparative study of regression vs pairwise models for surrogate-based heuristic optimisation

Enhancing Cooperative Coevolution for Large Scale Optimization by Adaptively Constructing Surrogate Models

Surrogate Model Assisted Cooperative Coevolution for Large Scale Optimization

CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios

Measuring Code Efficiency Optimization Capabilities with ACEOB

CoDBench: A Critical Evaluation of Data-driven Models for Continuous Dynamical Systems

Benchmarking a Benchmark: How Reliable is MS-COCO?

Ensemble of surrogates in black-box-type engineering optimization: Recent advances and applications

A Surrogate Model Assisted Evolutionary Algorithm for Computationally Expensive Design Optimization Problems with Discrete Variables.

A large-scale continuous optimization benchmark suite with versatile coupled heterogeneous modules

Surrogate-Based Simulation Optimization

Execution-Based Evaluation for Open-Domain Code Generation