Abstract:Predicting the performance of an optimization algorithm on a new problem instance is crucial in order to select the most appropriate algorithm for solving that problem instance. For this purpose, recent studies learn a supervised machine learning (ML) model using a set of problem landscape features linked to the performance achieved by the optimization algorithm. However, these models are black-box with the only goal of achieving good predictive performance, without providing explanations which landscape features contribute the most to the prediction of the performance achieved by the optimization algorithm. In this study, we investigate the expressiveness of problem landscape features utilized by different supervised ML models in automated algorithm performance prediction. The experimental results point out that the selection of the supervised ML method is crucial, since different supervised ML regression models utilize the problem landscape features differently and there is no common pattern with regard to which landscape features are the most informative.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: in the performance prediction of automated algorithms, how to interpret and understand the utilization of problem instance features (i.e., exploratory landscape analysis (ELA) features) by different supervised machine learning (ML) models. Specifically, the researchers hope to reveal through this study which ELA features contribute the most to the performance prediction of optimization algorithms, and whether the importance of these features varies with the ML models used.
### Problem Background
In the tasks of automated algorithm selection and configuration, predicting the performance of an optimization algorithm on a new problem instance is crucial. To achieve this goal, existing methods usually train a supervised machine - learning model using a set of problem landscape features related to the performance of the optimization algorithm. However, these models are often black - box models and cannot explain the specific impact of each ELA feature on the final performance prediction.
### Research Motivation
Although previous studies have shown that different supervised ML regression models will produce different results on the same learning task, there is currently a lack of research on the performance patterns of these models under different ML methods. In particular, it has not been clearly pointed out which ELA features are the most informative, and whether the importance of these features will change with different ML models. Therefore, this study aims to fill this gap and provide a deeper understanding of the importance of ELA features.
### Main Contributions
1. **Variation in Feature Importance**: The study found that the importance of ELA features varies with different supervised ML methods, and their expressiveness is questionable in the performance prediction of automated algorithms.
2. **Dependence Analysis**: The importance of features depends on the problem instance to be solved, the optimization algorithm running on this instance, and the supervised ML method used to learn the prediction model.
3. **Model Consistency Verification**: Through t - SNE visualization, the model consistency of the same ML algorithm on different data folds was verified, further confirming the similarity of the importance of ELA features.
### Method Overview
- **Dataset**: Use 24 single - objective continuous optimization problems in the COCO benchmark platform, each problem contains 50 instances, for a total of 1200 problem instances.
- **Feature Calculation**: Use the R package "flacco" to calculate ELA features, and 99 features were selected for analysis.
- **Performance Data**: Use a random configuration of modular CMA - ES as performance data, run each problem instance 10 times with a budget of 50,000 function evaluations.
- **Regression Models**: Test three regression models, namely decision tree (DT), random forest (RF), and deep neural network (DNN), and conduct experiments in single - objective regression (STR) and multi - objective regression (MTR) scenarios respectively.
- **Evaluation Metrics**: Use 50 - fold cross - validation and calculate the mean squared error (MAE) to evaluate model performance.
### Results Discussion
The experimental results show that different ML models have their own advantages and disadvantages in predicting the original target accuracy and its logarithmic transformation. In particular, in DT and RF models, some features such as `elacurv.grad_norm.min` show high importance in multiple scenarios. In addition, through SHAP value analysis, the researchers were able to explain the impact of each ELA feature on the prediction results at the global and local levels.
In summary, this study not only reveals the performance differences of different ML models in the performance prediction of automated algorithms, but also provides valuable insights into the importance of ELA features, laying the foundation for further theoretical research and practical applications.