Abstract:Algorithm selection, a critical process of automated machine learning, aims to identify the most suitable algorithm for solving a specific problem prior to execution. Mainstream algorithm selection techniques heavily rely on problem features, while the role of algorithm features remains largely unexplored. Due to the intrinsic complexity of algorithms, effective methods for universally extracting algorithm information are lacking. This paper takes a significant step towards bridging this gap by introducing Large Language Models (LLMs) into algorithm selection for the first time. By comprehending the code text, LLM not only captures the structural and semantic aspects of the algorithm, but also demonstrates contextual awareness and library function understanding. The high-dimensional algorithm representation extracted by LLM, after undergoing a feature selection module, is combined with the problem representation and passed to the similarity calculation module. The selected algorithm is determined by the matching degree between a given problem and different algorithms. Extensive experiments validate the performance superiority of the proposed model and the efficacy of each key module. Furthermore, we present a theoretical upper bound on model complexity, showcasing the influence of algorithm representation and feature selection modules. This provides valuable theoretical guidance for the practical implementation of our method.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to select algorithms that are most suitable for specific problems more effectively in automated machine learning. Specifically, existing algorithm selection techniques mainly rely on problem features, while the features of the algorithms themselves are largely ignored. Due to the inherent complexity of algorithms, there is a lack of generally applicable algorithm information extraction methods. Therefore, this paper fills this gap by introducing large - language models (LLMs), using LLMs to capture the structural, semantic, and contextual information of algorithms from code texts, thereby providing a more comprehensive algorithm representation. This method can not only enhance the accuracy of algorithm selection but also improve the generalization ability for new algorithms. ### Core Contributions of the Paper 1. **First Application of LLMs in Algorithm Selection**: This paper applies LLMs to the algorithm selection task for the first time, uses their powerful representation capabilities to extract discriminative algorithm features, and introduces an algorithm feature selection module to identify key features. 2. **Comprehensive Algorithm Representation**: The AS - LLM model demonstrates its advantages in the following three aspects: - Model the bidirectional nature of the algorithm selection task more meticulously. - Have generalization ability for new algorithms not encountered in training. - Show stronger performance advantages in different scenarios. 3. **Theoretical Verification**: This paper not only highlights the performance advantages of AS - LLM through empirical research but also provides a strict upper bound on the model complexity, providing a theoretical basis for model design and practical applications. ### Method Overview - **Algorithm Representation Module**: Use pre - trained LLMs to extract high - dimensional algorithm representations from code or description texts. These representations are processed by LSTM and then generate the final algorithm features through a linear layer. - **Feature Selection Module**: Since the features extracted by LLMs have a high dimension and not all features are relevant to the algorithm selection task, a feature selection module is introduced. Samples are generated through the Gumbel distribution, making the feature selection operation differentiable, thereby selecting the most relevant features. - **Similarity Calculation Module**: After processing the problem features and algorithm features through the multi - layer perceptron (MLP) module respectively, calculate the cosine similarity between them. Finally, an MLP layer synthesizes the similarity and other features to predict the matching degree between the problem and the algorithm. ### Theoretical Analysis - **Upper Bound of Model Complexity**: This paper derives the upper bound of the AS - LLM model complexity by introducing the concept of inductive Rademacher complexity. This theoretical analysis not only helps to evaluate the generalization ability of the model but also provides guidance for model selection, complexity control, and algorithm design. ### Conclusion This paper proposes an algorithm selection model AS - LLM based on LLMs. Through comprehensive algorithm representation and feature selection, it significantly improves the accuracy and generalization ability of algorithm selection. Theoretical analysis further verifies the effectiveness of this model.

Large Language Model-Enhanced Algorithm Selection: Towards Comprehensive Algorithm Representation

A Systematic Survey on Large Language Models for Algorithm Design

SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models

Executing Natural Language-Described Algorithms with Large Language Models: An Investigation

The Efficiency Spectrum of Large Language Models: An Algorithmic Survey

Algorithm Evolution Using Large Language Model.

When Large Language Model Meets Optimization

Large Language Models as Surrogate Models in Evolutionary Algorithms: A Preliminary Study

On the Design and Analysis of LLM-Based Algorithms

Unlock the Power of Algorithm Features: A Generalization Analysis for Algorithm Selection

LLM-Select: Feature Selection with Large Language Models

Feature Selection Based on a Large-Scale Many-Objective Evolutionary Algorithm

Exploring Large Language Models for Feature Selection: A Data-centric Perspective

Adaptive-Solver Framework for Dynamic Strategy Selection in Large Language Model Reasoning

LLM4AD: A Platform for Algorithm Design with Large Language Model

Large Language Models Synergize with Automated Machine Learning

Large Language Model Alignment: A Survey

Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models