Abstract:Algorithms often have tunable parameters that impact performance metrics such as runtime and solution quality. For many algorithms used in practice, no parameter settings admit meaningful worst-case bounds, so the parameters are made available for the user to tune. Alternatively, parameters may be tuned implicitly within the proof of a worst-case approximation ratio or runtime bound. Worst-case instances, however, may be rare or nonexistent in practice. A growing body of research has demonstrated that a data-driven approach to parameter tuning can lead to significant improvements in performance. This approach uses a training set of problem instances sampled from an unknown, application-specific distribution and returns a parameter setting with strong average performance on the training set. We provide techniques for deriving generalization guarantees that bound the difference between the algorithm’s average performance over the training set and its expected performance on the unknown distribution. Our results apply no matter how the parameters are tuned, be it via an automated or manual approach. The challenge is that for many types of algorithms, performance is a volatile function of the parameters: slightly perturbing the parameters can cause a large change in behavior. Prior research [e.g., 12, 16, 20, 62] has proved generalization bounds by employing case-by-case analyses of greedy algorithms, clustering algorithms, integer programming algorithms, and selling mechanisms. We streamline these analyses with a general theorem that applies whenever an algorithm’s performance is a piecewise-constant, piecewise-linear, or—more generally— piecewise-structured function of its parameters. Our results, which are tight up to logarithmic factors in the worst case, also imply novel bounds for configuring dynamic programming algorithms from computational biology.

Unlock the Power of Algorithm Features: A Generalization Analysis for Algorithm Selection

Feature Selection and Parameter Optimization for Support Vector Machines: A New Approach Based on Genetic Algorithm with Feature Chromosomes.

Generalization in portfolio-based algorithm selection

Large Language Model-Enhanced Algorithm Selection: Towards Comprehensive Algorithm Representation

Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regression

A Feature Selection Method Based on Feature Grouping and Genetic Algorithm

How Much Data Is Sufficient to Learn High-Performing Algorithms?

Generalization Ability of Feature-based Performance Prediction Models: A Statistical Analysis across Benchmarks

Mixed Feature Selection Based on Granulation and Approximation

Benchmarking Feature-based Algorithm Selection Systems for Black-box Numerical Optimization

An Opposition-Based Great Wall Construction Metaheuristic Algorithm With Gaussian Mutation for Feature Selection

Learning Algorithm Generalization Error Bounds via Auxiliary Distributions

A Survey of Meta-features Used for Automated Selection of Algorithms for Black-box Single-objective Continuous Optimization

A General Wrapper Approach to Selection of Class-Dependent Features

Feature Selection for Optimized High-Dimensional Biomedical Data Using an Improved Shuffled Frog Leaping Algorithm

A Surrogate-Assisted Evolutionary Algorithm with Random Feature Selection for Large-Scale Expensive Problems

Extreme Algorithm Selection With Dyadic Feature Representation

Towards a Theoretical Framework of Out-of-Distribution Generalization

GENERALIZATION BOUNDS OF REGULARIZATION ALGORITHMS DERIVED SIMULTANEOUSLY THROUGH HYPOTHESIS SPACE COMPLEXITY, ALGORITHMIC STABILITY AND DATA QUALITY

A hybrid genetic algorithm for feature selection wrapper based on mutual information

A Novel Hybrid Algorithm for Feature Selection