Abstract:Selection hyper-heuristics (HHs) are randomised search methodologies which choose and execute heuristics during the optimisation process from a set of low-level heuristics. A machine learning mechanism is generally used to decide which low-level heuristic should be applied in each decision step. In this article, we analyse whether sophisticated learning mechanisms are always necessary for HHs to perform well. To this end we consider the most simple HHs from the literature and rigorously analyse their performance for the LeadingOnes benchmark function. Our analysis shows that the standard Simple Random, Permutation, Greedy, and Random Gradient HHs show no signs of learning. While the former HHs do not attempt to learn from the past performance of low-level heuristics, the idea behind the Random Gradient HH is to continue to exploit the currently selected heuristic as long as it is successful. Hence, it is embedded with a reinforcement learning mechanism with the shortest possible memory. However, the probability that a promising heuristic is successful in the next step is relatively low when perturbing a reasonable solution to a combinatorial optimisation problem. We generalise the "simple" Random Gradient HH so success can be measured over a fixed period of time τ, instead of a single iteration. For LeadingOnes we prove that the Generalised Random Gradient (GRG) HH can learn to adapt the neighbourhood size of Randomised Local Search to optimality during the run. As a result, we prove it has the best possible performance achievable with the low-level heuristics (Randomised Local Search with different neighbourhood sizes), up to lower-order terms. We also prove that the performance of the HH improves as the number of low-level local search heuristics to choose from increases. In particular, with access to k low-level local search heuristics, it outperforms the best-possible algorithm using any subset of the k heuristics. Finally, we show that the advantages of GRG over Randomised Local Search and Evolutionary Algorithms using standard bit mutation increase if the anytime performance is considered (i.e., the performance gap is larger if approximate solutions are sought rather than exact ones). Experimental analyses confirm these results for different problem sizes (up to n=108) and shed some light on the best choices for the parameter τ in various situations.

A review of reinforcement learning based hyper-heuristics

Combining Reinforcement Learning with Mathematical Programming: an Approach for Optimal Design of Heat Exchanger Networks

Reinforcement Learning Driven Heuristic Optimization

A Hierarchical Reinforcement Learning-Aware Hyper-Heuristic Algorithm with Fitness Landscape Analysis

Summarize of hierarchical reinforcement learning

Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization

Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards

Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF

Hyper-heuristics: a survey of the state of the art

Hierarchical Reinforcement Learning: A Survey and Open Research Challenges

A Review of Reinforcement Learning in Financial Applications

HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem

Efficient hyperparameters optimization through model-based reinforcement learning with experience exploiting and meta-learning

Large-State Reinforcement Learning for Hyper-Heuristics

Hybrid Reinforcement Learning Framework for Mixed-Variable Problems

Simple Hyper-Heuristics Control the Neighbourhood Size of Randomised Local Search Optimally for LeadingOnes

Hyperparameter Optimization for Multi-Objective Reinforcement Learning

HeuRL: A Heuristically Initialized Reinforcement Learning Method for Autonomous Driving Control Task

Enhanced Harris hawks optimization with multi-strategy for global optimization tasks

Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policies

A Survey on Applications of Reinforcement Learning in Spatial Resource Allocation