Comparison of Bayesian and particle swarm algorithms for hyperparameter optimisation in machine learning applications in high energy physics

Laurits Tani,Christian Veelken

DOI: https://doi.org/10.1016/j.cpc.2023.108955

2023-10-13

Abstract:When using machine learning (ML) techniques, users typically need to choose a plethora of algorithm-specific parameters, referred to as hyperparameters. In this paper, we compare the performance of two algorithms, particle swarm optimisation (PSO) and Bayesian optimisation (BO), for the autonomous determination of these hyperparameters in applications to different ML tasks typical for the field of high energy physics (HEP). Our evaluation of the performance includes a comparison of the capability of the PSO and BO algorithms to make efficient use of the highly parallel computing resources that are characteristic of contemporary HEP experiments.

Data Analysis, Statistics and Probability,High Energy Physics - Experiment

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to compare the hyper - parameter optimization performance of two algorithms in machine learning (ML) tasks in the field of high - energy physics (HEP). Specifically, the paper compares the performance of the Particle Swarm Optimization (PSO) and Bayesian Optimization (BO) algorithms in autonomously determining hyper - parameters. The research focuses include:

1. **Hyper - parameter optimization**: Machine learning algorithms usually require users to select a series of algorithm - specific parameters, called hyper - parameters. The selection of these hyper - parameters has a great impact on the algorithm performance. The paper explores how to use PSO and BO algorithms to efficiently determine these hyper - parameters.

2. **Utilization of parallel computing resources**: High - energy physics experiments usually need to process a large amount of data, and a single training may take hours or even days. Therefore, efficient utilization of large - scale parallel computing resources is the key to improving the efficiency of hyper - parameter optimization. The paper evaluates the performance of PSO and BO algorithms in a parallel computing environment.

3. **Benchmark test tasks**: In order to evaluate the performance of these two algorithms, the paper selects two benchmark test tasks:
- **The problem of minimizing the Rosenbrock function**: This is a commonly used test function for evaluating the performance of function minimization algorithms.
- **The ATLAS Higgs boson machine learning challenge**: This is a typical high - energy physics data analysis task, which involves using machine learning algorithms to classify signal and background events.

Through these benchmark test tasks, the paper aims to answer the following questions:
- Which algorithm performs better in different numbers of hyper - parameter evaluations?
- How do these algorithms perform on parallel computing resources?
- How does the computational overhead of the algorithm affect its performance?

In general, this paper attempts to provide a basis for researchers in the field of high - energy physics to select appropriate optimization algorithms by comparing the performance of PSO and BO algorithms in hyper - parameter optimization.