Quality Indicators for Preference-based Evolutionary Multi-objective Optimization Using a Reference Point: A Review and Analysis

Ryoji Tanabe,Ke Li
DOI: https://doi.org/10.1109/TEVC.2023.3319009
2023-09-26
Abstract:Some quality indicators have been proposed for benchmarking preference-based evolutionary multi-objective optimization algorithms using a reference point. Although a systematic review and analysis of the quality indicators are helpful for both benchmarking and practical decision-making, neither has been conducted. In this context, first, this paper reviews existing regions of interest and quality indicators for preference-based evolutionary multi-objective optimization using the reference point. We point out that each quality indicator was designed for a different region of interest. Then, this paper investigates the properties of the quality indicators. We demonstrate that an achievement scalarizing function value is not always consistent with the distance from a solution to the reference point in the objective space. We observe that the regions of interest can be significantly different depending on the position of the reference point and the shape of the Pareto front. We identify undesirable properties of some quality indicators. We also show that the ranking of preference-based evolutionary multi-objective optimization algorithms depends on the choice of quality indicators.
Neural and Evolutionary Computing
What problem does this paper attempt to address?
### What problem does this paper attempt to solve? This paper primarily explores the issue of quality indicators in Preference-based Evolutionary Multi-objective Optimization (PBEMO) and provides a systematic review and analysis of these indicators. #### Main Issues: 1. **Review of Quality Indicators**: The paper reviews existing Region of Interest (ROI) and reference point-based quality indicators, highlighting that each quality indicator focuses on different ROIs during its design. 2. **Analysis of Quality Indicator Properties**: The study examines the properties of quality indicators, finding that the Achievement Scalarizing Function (ASF) values do not always align with the distance from the solution to the reference point. Additionally, the definition of different ROIs can affect the behavior of EMO algorithms. 3. **Shortcomings of Quality Indicators**: The paper identifies some undesirable characteristics of quality indicators and points out that choosing different quality indicators can lead to varying results in the performance evaluation of PBEMO algorithms. 4. **Impact of Quality Indicator Selection**: It demonstrates how the selection of different quality indicators can influence the ranking of PBEMO algorithms. #### Research Motivation: - **Review Motivation**: Although several preference-based quality indicators have been proposed, there is a lack of systematic investigation. - **Analysis Motivation**: The properties of quality indicators are not obvious, including which point sets they prefer and whether there is consistency among different indicators. Therefore, using specific quality indicators might incorrectly assess the performance of EMO algorithms. #### Contributions: - **Clarification of ROI and Quality Indicators**: The paper categorizes quality indicators based on the target ROI and clearly identifies the target ROI for each quality indicator. - **Analysis of Quality Indicators**: By analyzing, the paper answers four research questions, including whether the Pareto optimal point with the minimum ASF value always minimizes the distance to the reference point, the differences in the definition of different ROIs and their impact on EMO algorithm behavior, the characteristics of existing quality indicators, and the impact of quality indicator selection on the ranking of PBEMO algorithms. Through the above work, the paper aims to provide a deeper understanding of the performance evaluation of PBEMO algorithms and offer guidance for future research.