Abstract:We introduce a novel LLM based solution design approach that utilizes combinatorial optimization and sampling. Specifically, a set of factors that influence the quality of the solution are identified. They typically include factors that represent prompt types, LLM inputs alternatives, and parameters governing the generation and design alternatives. Identifying the factors that govern the LLM solution quality enables the infusion of subject matter expert knowledge. Next, a set of interactions between the factors are defined and combinatorial optimization is used to create a small subset $P$ that ensures all desired interactions occur in $P$. Each element $p \in P$ is then developed into an appropriate benchmark. Applying the alternative solutions on each combination, $p \in P$ and evaluating the results facilitate the design of a high quality LLM solution pipeline. The approach is especially applicable when the design and evaluation of each benchmark in $P$ is time-consuming and involves manual steps and human evaluation. Given its efficiency the approach can also be used as a baseline to compare and validate an autoML approach that searches over the factors governing the solution.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve The paper aims to address the issue of how to design high-quality large-scale language model (LLM) solutions. Specifically, the authors propose a new method based on combinatorial optimization and sampling to design LLM solutions. This method identifies factors that affect the quality of the solution, defines the interactions between these factors, and uses combinatorial optimization techniques to generate a small test set that ensures all desired interactions are included. Each element of the test set is then developed into appropriate benchmarks, and by applying different solutions and evaluating the results, a high-quality LLM solution pipeline is designed. ### Main Contributions 1. **Systematic Integration of Human Knowledge**: Systematically integrating domain experts' knowledge into the benchmark design. 2. **Automated Assistance in Defining Design Space and Coverage**: Providing automated tools to define the design space and its coverage. 3. **Efficient Sampling of Important Variations in Experiments**: Offering techniques to efficiently sample the most important variations in the design space. 4. **Identification of Key Factors**: Providing a method to identify the factors that contribute most to the quality of results with minimal data in specific experimental settings. ### Key Steps of the Solution 1. **Exploration Phase**: Exploring available models by trying different prompts, inputs, and generation parameters to identify factors affecting solution quality. 2. **Analysis Phase**: Developing benchmark data and metrics to determine the quality of LLM solutions and further development. 3. **Control Phase**: Monitoring the performance of LLM solutions to ensure they work as expected. 4. **Regression Phase**: Re-evaluating and analyzing benchmarks, which may be triggered by new LLM model candidates or changes in usage patterns. ### Combinatorial Optimization Problem The paper details how to handle the combinatorial explosion of the design space using combinatorial optimization techniques. Specifically, by selecting low-order interactions (such as 1st-order, 2nd-order, or 3rd-order interactions), a small test set is generated to ensure all important interactions are covered. Then, statistical analysis methods (such as analysis of variance) are used to identify the parameters that have the greatest impact on solution quality. ### Case Study The paper illustrates this process through a specific case (using LLM to generate code summaries). In this case, the authors identified several factors affecting the quality of code summaries, including prompt type, the quality of input code documentation, generation parameters, etc. By generating a test set through combinatorial optimization, the authors were able to evaluate the effects of different parameter combinations and select the best solution. ### Conclusion The method proposed in the paper not only systematically designs high-quality LLM solutions but also efficiently revalidates or adjusts parameter selections when new LLM models emerge. This method is particularly suitable for designing and evaluating benchmarks that are time-consuming and partially manual.

Using Combinatorial Optimization to Design a High quality LLM Solution

LLM as a Complementary Optimizer to Gradient Descent: A Case Study in Prompt Tuning

Large Language Models for Combinatorial Optimization of Design Structure Matrix

Search-Based LLMs for Code Optimization

OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling

LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch

LLMs are Highly-Constrained Biophysical Sequence Optimizers

AIME: AI System Optimization via Multiple LLM Evaluators

Finetuning LLMs for Comparative Assessment Tasks

Learning for Robust Combinatorial Optimization: Algorithm and Application

Actively Learning Combinatorial Optimization Using a Membership Oracle

Landscape-Sketch-Step: An AI/ML-Based Metaheuristic for Surrogate Optimization Problems

Addressing Data Leakage in HumanEval Using Combinatorial Test Design

Combining Reinforcement Learning and Constraint Programming for Combinatorial Optimization

ADO-LLM: Analog Design Bayesian Optimization with In-Context Learning of Large Language Models

Accelerating Multi-Block Constrained Optimization Through Learning to Optimize

Solving General Natural-Language-Description Optimization Problems with Large Language Models

In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics

Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons

Exploring the Capabilities of Large Language Models for Generating Diverse Design Solutions