Abstract:In scientific disciplines, benchmarks play a vital role in driving progress forward. For a benchmark to be effective, it must closely resemble real-world tasks. If the level of difficulty or relevance is inadequate, it can impede progress in the field. Moreover, benchmarks should have low computational overhead to ensure accessibility and repeatability. The objective is to achieve a kind of ``Turing test'' by creating a surrogate model that is practically indistinguishable from the ground truth observation, at least within the dataset's explored boundaries. This objective necessitates a large quantity of data. This data encompasses numerous features that are characteristic of chemistry and materials science optimization tasks that are relevant to industry. These features include high levels of noise, multiple fidelities, multiple objectives, linear constraints, non-linear correlations, and failure regions. We performed 494498 random hard-sphere packing simulations representing 206 CPU days' worth of computational overhead. Simulations required nine input parameters with linear constraints and two discrete fidelities each with continuous fidelity parameters. The data was logged in a free-tier shared MongoDB Atlas database, producing two core tabular datasets: a failure probability dataset and a regression dataset. The failure probability dataset maps unique input parameter sets to the estimated probabilities that the simulation will fail. The regression dataset maps input parameter sets (including repeats) to particle packing fractions and computational runtimes for each of the two steps. These two datasets were used to create a surrogate model as close as possible to running the actual simulations by incorporating simulation failure and heteroskedastic noise. In the regression dataset, percentile ranks were calculated for each group of identical parameter sets to account for heteroskedastic noise, thereby ensuring reliable and accurate data. This differs from the conventional approach that imposes a-priori assumptions, such as Gaussian noise, by specifying mean and standard deviation. This technique can be extended to other benchmark datasets to bridge the gap between optimization benchmarks with low computational overhead and the complex optimization scenarios encountered in the real world.

Working with benchmark datasets in the Cuby framework

Working with benchmark datasets in the Cuby framework

Codabench: Flexible, Easy-to-use, and Reproducible Meta-Benchmark Platform

Benchmarks for Characterization of Minima, Transition States, and Pathways in Atomic, Molecular, and Condensed Matter Systems.

Benchmark Frameworks and Τbench

A Continuous Benchmarking Infrastructure for High-Performance Computing Applications

A framework for benchmarking clustering algorithms

Building a continuous benchmarking ecosystem in bioinformatics

Quantum Monte Carlo for Noncovalent Interactions: An Efficient Protocol Attaining Benchmark Accuracy

Autobench V1.0: Benchmarking Automation for Electronic Structure Calculations

Omnibenchmark (alpha) for continuous and open benchmarking in bioinformatics

Benchpress: A Scalable and Versatile Workflow for Benchmarking Structure Learning Algorithms

A New Angle on Benchmarking Noncovalent Interactions

Materials science optimization benchmark dataset for multi-objective, multi-fidelity optimization of hard-sphere packing simulations

Tartarus: A Benchmarking Platform for Realistic And Practical Inverse Molecular Design

CSPBench: a benchmark and critical evaluation of Crystal Structure Prediction

Benchmarking Quantum Computer Simulation Software Packages: State Vector Simulators

CatTestHub: A Benchmarking Database of Experimental Heterogeneous Catalysis for Evaluating Advanced Materials

Quantum chemical benchmark databases of gold-standard dimer interaction energies

HydroNet: Benchmark Tasks for Preserving Intermolecular Interactions and Structural Motifs in Predictive and Generative Models for Molecular Data

A Modular Workflow for Performance Benchmarking of Neuronal Network Simulations