Abstract:In scientific disciplines, benchmarks play a vital role in driving progress forward. For a benchmark to be effective, it must closely resemble real-world tasks. If the level of difficulty or relevance is inadequate, it can impede progress in the field. Moreover, benchmarks should have low computational overhead to ensure accessibility and repeatability. The objective is to achieve a kind of ``Turing test'' by creating a surrogate model that is practically indistinguishable from the ground truth observation, at least within the dataset's explored boundaries. This objective necessitates a large quantity of data. This data encompasses numerous features that are characteristic of chemistry and materials science optimization tasks that are relevant to industry. These features include high levels of noise, multiple fidelities, multiple objectives, linear constraints, non-linear correlations, and failure regions. We performed 494498 random hard-sphere packing simulations representing 206 CPU days' worth of computational overhead. Simulations required nine input parameters with linear constraints and two discrete fidelities each with continuous fidelity parameters. The data was logged in a free-tier shared MongoDB Atlas database, producing two core tabular datasets: a failure probability dataset and a regression dataset. The failure probability dataset maps unique input parameter sets to the estimated probabilities that the simulation will fail. The regression dataset maps input parameter sets (including repeats) to particle packing fractions and computational runtimes for each of the two steps. These two datasets were used to create a surrogate model as close as possible to running the actual simulations by incorporating simulation failure and heteroskedastic noise. In the regression dataset, percentile ranks were calculated for each group of identical parameter sets to account for heteroskedastic noise, thereby ensuring reliable and accurate data. This differs from the conventional approach that imposes a-priori assumptions, such as Gaussian noise, by specifying mean and standard deviation. This technique can be extended to other benchmark datasets to bridge the gap between optimization benchmarks with low computational overhead and the complex optimization scenarios encountered in the real world.

Towards Realistic Optimization Benchmarks: A Questionnaire on the Properties of Real-World Problems

Identifying Properties of Real-World Optimisation Problems through a Questionnaire

Benchmarking Real-World Many-Objective Problems: A Problem Suite With Baseline Results

AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite

On the Assessment of Benchmark Suites for Algorithm Comparison

Benchmarking for Metaheuristic Black-Box Optimization: Perspectives and Open Challenges

Scalable and customizable benchmark problems for many-objective optimization

Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis

OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling

Towards Robust Benchmarking of Quantum Optimization Algorithms

Materials science optimization benchmark dataset for multi-objective, multi-fidelity optimization of hard-sphere packing simulations

Benchmarking Meta-heuristic Optimization

Diagnostic benchmarking of many-objective evolutionary algorithms for real-world problems

Offline Multi-Objective Optimization

A Tutorial on the Design, Experimentation and Application of Metaheuristic Algorithms to Real-World Optimization Problems

Bridging the Gap Between Theory and Practice: Benchmarking Transfer Evolutionary Optimization

Benchmarking the performance of Bayesian optimization across multiple experimental materials science domains

On benchmarking and good scientific practise in topology optimization

Comparative Analysis of Selection Hyper-Heuristics for Real-World Multi-Objective Optimization Problems

Benchmarking multiobjective evolutionary algorithms and constraint handling techniques on a real-world car structure design optimization benchmark problem

A benchmark generator for scenario-based discrete optimization