On the Fair Comparison of Optimization Algorithms in Different Machines
Etor Arza,Josu Ceberio,Ekhiñe Irurozki,Aritz Pérez
DOI: https://doi.org/10.1214/23-AOAS1778
2023-05-12
Abstract:An experimental comparison of two or more optimization algorithms requires
the same computational resources to be assigned to each algorithm. When a
maximum runtime is set as the stopping criterion, all algorithms need to be
executed in the same machine if they are to use the same resources.
Unfortunately, the implementation code of the algorithms is not always
available, which means that running the algorithms to be compared in the same
machine is not always possible. And even if they are available, some
optimization algorithms might be costly to run, such as training large
neural-networks in the cloud.
In this paper, we consider the following problem: how do we compare the
performance of a new optimization algorithm B with a known algorithm A in the
literature if we only have the results (the objective values) and the runtime
in each instance of algorithm A? Particularly, we present a methodology that
enables a statistical analysis of the performance of algorithms executed in
different machines. The proposed methodology has two parts. First, we propose a
model that, given the runtime of an algorithm in a machine, estimates the
runtime of the same algorithm in another machine. This model can be adjusted so
that the probability of estimating a runtime longer than what it should be is
arbitrarily low. Second, we introduce an adaptation of the one-sided sign test
that uses a modified p-value and takes into account that probability. Such
adaptation avoids increasing the probability of type I error associated with
executing algorithms A and B in different machines.
Optimization and Control,Applications,Data Structures and Algorithms,Performance