Abstract:Hyperparameter tuning is an active area of research in machine learning, where the aim is to identify the optimal hyperparameters that provide the best performance on the validation set. Hyperparameter tuning is often achieved using naive techniques, such as random search and grid search. However, most of these methods seldom lead to an optimal set of hyperparameters and often get very expensive. In this paper, we propose a bilevel solution method for solving the hyperparameter optimization problem that does not suffer from the drawbacks of the earlier studies. The proposed method is general and can be easily applied to any class of machine learning algorithms. The idea is based on the approximation of the lower level optimal value function mapping, which is an important mapping in bilevel optimization and helps in reducing the bilevel problem to a single level constrained optimization task. The single-level constrained optimization problem is solved using the augmented Lagrangian method. We discuss the theory behind the proposed algorithm and perform extensive computational study on two datasets that confirm the efficiency of the proposed method. We perform a comparative study against grid search, random search and Bayesian optimization techniques that shows that the proposed algorithm is multiple times faster on problems with one or two hyperparameters. The computational gain is expected to be significantly higher as the number of hyperparameters increase. Corresponding to a given hyperparameter most of the techniques in the literature often assume a unique optimal parameter set that minimizes loss on the training set. Such an assumption is often violated by deep learning architectures and the proposed method does not require any such assumption.

Bi-level algorithm for optimizing hyperparameters in penalized nonnegative matrix factorization

Penalty hyperparameter optimization with diversity measure for nonnegative low-rank approximation

Optimization and expansion of non-negative matrix factorization

Relax and penalize: a new bilevel approach to mixed-binary hyperparameter optimization

NeNMF: an Optimal Gradient Method for Nonnegative Matrix Factorization.

Efficient algorithms for regularized Poisson Non-negative Matrix Factorization

Theoretical aspects in penalty hyperparameters optimization

Using Population Based Algorithms For Initializing Nonnegative Matrix Factorization

Alternating block linearized Bregman iterations for regularized nonnegative matrix factorization

Alternating optimization method based on nonnegative matrix factorizations for deep neural networks

Swarm Intelligence For Non-Negative Matrix Factorization

An Adaptive Nonmonotone Projected Barzilai-Borwein Gradient Method with Active Set Prediction for Nonnegative Matrix Factorization

Adaptive computation of the Symmetric Nonnegative Matrix Factorization (NMF)

Blind Matrix Decomposition Via Genetic Optimization Of Sparseness And Nonnegativity Constraints

Efficient Hyper-parameter Optimization for NLP Applications.

Bayesian mean-parameterized nonnegative binary matrix factorization

A Two-Phase Algorithm for Robust Symmetric Non-Negative Matrix Factorization

A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning

A Gradient-Based Algorithm with Nonmonotone Line Search for Nonnegative Matrix Factorization

An effective algorithm for hyperparameter optimization of neural networks

Deep Alternating Non-Negative Matrix Factorisation