Abstract:No Free Lunch Theorem presents a dilemma in the evaluation of emerging <a class="topic-link" href="/topics/mathematics/evolutionary-algorithm">evolutionary algorithms</a> in terms of handling various <a class="topic-link" href="/topics/computer-science/real-world-problem">real world problems</a> and their unknown internal structures, since the performances of these algorithms are related to the corresponding benchmarks. Although white and black box schemes have made impressive progress in overcoming this dilemma, such as clear <a class="topic-link" href="/topics/mathematics/sigma-property">property</a> definition and basis function composition, the evaluation of algorithms on sophisticated suites remains insufficient on account of the limited quantity and diversity of such benchmarks, which can induce bias in a narrow problem domain. Therefore, this study proposes a novel framework for randomly generating diversified <a class="topic-link" href="/topics/computer-science/benchmark-function">benchmark functions</a> to comprehensively evaluate evolutionary algorithms in a black box scenario. The proposed approach adopts a <a class="topic-link" href="/topics/engineering/recurrent-neural-network">recurrent neural network</a> with various <a class="topic-link" href="/topics/computer-science/activation-function">activation functions</a> to produce test problems with important characteristics such as ruggedness and multi-funnels. In addition, the proposed framework can generate virtually limitless chaotic benchmarks by using random weights. The experimental results demonstrate a distinct difference among the performance of the tested optimizers on the proposed problems and the well-known BBOB and CEC problems, which implies the necessity of the proposed benchmarks when facilitating a more comprehensive evaluation.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges faced in evaluating emerging evolutionary algorithms in black - box optimization. Specifically, due to the limited number and insufficient diversity of existing benchmark problems, the evaluation of algorithm performance may be biased, especially when dealing with real - world problems with unknown internal structures. Therefore, the paper proposes a new framework for randomly generating diverse benchmark functions to comprehensively evaluate evolutionary algorithms in black - box scenarios. ### Main Problems 1. **Limitations of Existing Benchmarks**: - **Limited Quantity**: The number of existing benchmark problems is limited and cannot fully cover all possible problem types. - **Lack of Diversity**: The problem types and properties of existing benchmarks are relatively fixed and lack sufficient diversity, making it difficult to reflect the complexity of the real world. - **Potential Bias**: Due to the limitations of the number and type of problems, the evaluation results may be biased within specific problem domains. 2. **Challenges in Black - Box Optimization**: - **Unknown Internal Structure**: The internal structures of real - world problems are often unknown, which makes traditional white - box testing methods difficult to apply. - **Algorithm Adaptability**: In order to effectively solve various problems, optimization algorithms need to have stronger adaptability and robustness. ### Solutions The paper proposes a Chaotic Landscape Generator (CLG) based on Recurrent Neural Networks (RNN) to generate diverse benchmark problems. Main features include: - **Random Generation**: By randomly generating weights and using multiple activation functions, it generates benchmark problems with diversity and irregularity. - **Infinite Expansion**: It can generate an almost infinite number of chaotic benchmark problems, reducing bias in evaluation. - **Enhanced Diversity**: The generated problems have complex characteristics such as ruggedness and multimodality, which can more comprehensively evaluate the performance of optimization algorithms. ### Main Contributions 1. **Diverse Benchmark Generation**: A new framework is proposed that can randomly generate diverse benchmark problems for comprehensively evaluating optimization algorithms. 2. **Introduction of New Activation Functions**: Multiple nonlinear activation functions are used to enhance the diversity and complexity of the generated problems. ### Experimental Results The experimental results show that there are significant differences in the performance of optimizers on the proposed benchmark problems compared to the known BBOB and CEC problems, indicating the necessity of the new benchmark problems for more comprehensive evaluation of optimization algorithms. ### Conclusion This research proposes a new Chaotic Landscape Generator for generating diverse benchmark problems to comprehensively evaluate optimization algorithms in black - box scenarios. By randomly generating weights and using multiple activation functions, this method can generate benchmark problems with complex characteristics, thereby reducing bias in evaluation and improving the adaptability and robustness of optimization algorithms.

A neuro-diversified benchmark generator for black box optimization

An New Evolutionary Multi-Objective Optimization Algorithm

Aibench: an industry standard ai benchmark suite

MA-BBOB: A Problem Generator for Black-Box Optimization Using Affine Combinations and Shifts

Benchmarking Feature-based Algorithm Selection Systems for Black-box Numerical Optimization

Benchmarking for Metaheuristic Black-Box Optimization: Perspectives and Open Challenges

Impact of Training Instance Selection on Automated Algorithm Selection Models for Numerical Black-box Optimization

A benchmark generator for scenario-based discrete optimization

NeuroEvoBench: Benchmarking Evolutionary Optimizers for Deep Learning Applications

A large-scale continuous optimization benchmark suite with versatile coupled heterogeneous modules

Versatile Black-Box Optimization

B2Opt: Learning to Optimize Black-box Optimization with Little Budget

Evolving Benchmark Functions to Compare Evolutionary Algorithms via Genetic Programming

Bridging the Gap Between Theory and Practice: Benchmarking Transfer Evolutionary Optimization

Illuminating the Diversity-Fitness Trade-Off in Black-Box Optimization

BEACON: A Bayesian Optimization Strategy for Novelty Search in Expensive Black-Box Systems

Assessing Quality-Diversity Neuro-Evolution Algorithms Performance in Hard Exploration Problems

OpenBox: A Generalized Black-box Optimization Service

Benchmarking Derivative-Free Global Optimization Algorithms Under Limited Dimensions and Large Evaluation Budgets

NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems

Benchmarking Analysis of Evolutionary Neural Architecture Search