Abstract:Hyperparameter optimization in machine learning is often achieved using naive techniques that only lead to an approximate set of hyperparameters. Although techniques such as Bayesian optimization perform an intelligent search on a given domain of hyperparameters, it does not guarantee an optimal solution. A major drawback of most of these approaches is an exponential increase of their search domain with number of hyperparameters, increasing the computational cost and making the approaches slow. The hyperparameter optimization problem is inherently a bilevel optimization task, and some studies have attempted bilevel solution methodologies for solving this problem. However, these studies assume a unique set of model weights that minimize the training loss, which is generally violated by deep learning architectures. This paper discusses a gradient-based bilevel method addressing these drawbacks for solving the hyperparameter optimization problem. The proposed method can handle continuous hyperparameters for which we have chosen the regularization hyperparameter in our experiments. The method guarantees convergence to the set of optimal hyperparameters that this study has theoretically proven. The idea is based on approximating the lower-level optimal value function using Gaussian process regression. As a result, the bilevel problem is reduced to a single level constrained optimization task that is solved using the augmented Lagrangian method. We have performed an extensive computational study on the MNIST and CIFAR-10 datasets on multi-layer perceptron and LeNet architectures that confirms the efficiency of the proposed method. A comparative study against grid search, random search, Bayesian optimization, and HyberBand method on various hyperparameter problems shows that the proposed algorithm converges with lower computation and leads to models that generalize better on the testing set.

Implicit Differentiation for Hyperparameter Tuning the Weighted Graphical Lasso

A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning

Gradient-based bilevel optimization for multi-penalty Ridge regression through matrix differential calculus

An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning

Analyzing Inexact Hypergradients for Bilevel Learning

A Globally Convergent Gradient-based Bilevel Hyperparameter Optimization Method

Graduated Optimization of Black-Box Functions

Lower-level Duality Based Reformulation and Majorization Minimization Algorithm for Hyperparameter Optimization

Robust Implicit Regularization via Weight Normalization

A Generalized Alternating Method for Bilevel Learning under the Polyak-Łojasiewicz Condition

Implicit Iterative Algorithms with a Tuning Parameter for Discrete Stochastic Lyapunov Matrix Equations

Efficient Alternating Minimization with Applications to Weighted Low Rank Approximation

LancBiO: dynamic Lanczos-aided bilevel optimization via Krylov subspace

A first-order optimization algorithm for statistical learning with hierarchical sparsity structure

Double Momentum Method for Lower-Level Constrained Bilevel Optimization

Latent Gaussian Graphical Models with Golazo Penalty

Auptimizer -- an Extensible, Open-Source Framework for Hyperparameter Tuning

Practical Bayesian Optimization of Machine Learning Algorithms

Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm

On Finding Small Hyper-Gradients in Bilevel Optimization: Hardness Results and Improved Analysis

An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation