Abstract:Large Language Models (LLMs) have shown great potential in automatically generating and optimizing (meta)heuristics, making them valuable tools in heuristic optimization tasks. However, LLMs are generally inefficient when it comes to fine-tuning hyper-parameters of the generated algorithms, often requiring excessive queries that lead to high computational and financial costs. This paper presents a novel hybrid approach, LLaMEA-HPO, which integrates the open source LLaMEA (Large Language Model Evolutionary Algorithm) framework with a Hyper-Parameter Optimization (HPO) procedure in the loop. By offloading hyper-parameter tuning to an HPO procedure, the LLaMEA-HPO framework allows the LLM to focus on generating novel algorithmic structures, reducing the number of required LLM queries and improving the overall efficiency of the optimization process. We empirically validate the proposed hybrid framework on benchmark problems, including Online Bin Packing, Black-Box Optimization, and the Traveling Salesperson Problem. Our results demonstrate that LLaMEA-HPO achieves superior or comparable performance compared to existing LLM-driven frameworks while significantly reducing computational costs. This work highlights the importance of separating algorithmic innovation and structural code search from parameter tuning in LLM-driven code optimization and offers a scalable approach to improve the efficiency and effectiveness of LLM-based code generation.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the inefficiency of fine - tuning the hyper - parameters of the generated algorithms when large - language models (LLMs) generate and optimize heuristic algorithms. Specifically, LLMs usually require a large number of queries to adjust these hyper - parameters, which leads to high computational and financial costs. To solve this problem, the author proposes a new hybrid framework - LLaMEA - HPO, which combines the open - source LLaMEA framework with the hyper - parameter optimization (HPO) process. ### Main problems 1. **High computational and financial costs**: Existing LLM - driven automatic design methods require a large number of queries when optimizing the hyper - parameters of the generated algorithms, which leads to high costs. 2. **Inefficiency**: LLMs are relatively inefficient in fine - tuning hyper - parameters. Especially in evolutionary computation, LLMs often spend a great deal of time on hyper - parameter adjustment rather than generating novel algorithmic structures. 3. **Waste of resources**: Since LLMs are good at generating natural languages and codes but are insufficiently trained in numerical hyper - parameter settings, using LLMs for hyper - parameter adjustment is a waste of resources. ### Solutions To address the above problems, the paper proposes the LLaMEA - HPO framework, whose main features include: - **Task separation**: By assigning the hyper - parameter optimization task to a specialized HPO tool (such as SMAC), the LLM can focus on generating novel algorithmic structures and control flows, thereby improving overall efficiency and reducing unnecessary queries. - **Reduction in the number of queries**: By introducing the HPO tool, the number of queries to the LLM is reduced, and the computational and financial costs are lowered. - **Performance improvement**: The experimental results show that LLaMEA - HPO not only significantly reduces the computational cost but also achieves performance equivalent to or better than that of existing LLM - driven frameworks on multiple benchmark problems. ### Experimental verification The author verifies the effectiveness of LLaMEA - HPO on several benchmark problems, including online binary packing, black - box optimization, and the traveling salesman problem. The experimental results show that LLaMEA - HPO can maintain or even improve the algorithm performance while reducing the number of LLM queries. ### Summary The main contribution of this paper lies in proposing an innovative hybrid framework. By separating the hyper - parameter optimization task from the LLM, the efficiency and effectiveness of the LLM in generating and optimizing heuristic algorithms are improved. This method not only reduces the computational and financial costs but also provides valuable insights for future LLM - driven automated design.

In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics

Large Language Model Agent for Hyper-Parameter Optimization

Sequential Large Language Model-Based Hyper-Parameter Optimization

Using Large Language Models for Hyperparameter Optimization

LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics

Hyper-heuristics with low level parameter adaptation

Multi-objective Evolution of Heuristic Using Large Language Model

Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithms

Metaheuristics and Large Language Models Join Forces: Towards an Integrated Optimization Approach

Optimizing High-Level Synthesis Designs with Retrieval-Augmented Large Language Models

Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset

Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model

Large Language Models to Enhance Bayesian Optimization

Autonomous Multi-Objective Optimization Using Large Language Model

A Human Learning Optimization Algorithm and Its Application to Multi-Dimensional Knapsack Problems

Optimizing Large Language Models for Dynamic Constraints through Human-in-the-Loop Discriminators

Hyperparameter Optimization for Large Language Model Instruction-Tuning

An investigation on the use of Large Language Models for hyperparameter tuning in Evolutionary Algorithms

ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution

Efficient hyperparameters optimization through model-based reinforcement learning with experience exploiting and meta-learning