Bilevel Optimization for Automated Machine Learning: a New Perspective on Framework and Algorithm

Risheng Liu,Zhouchen Lin

DOI: https://doi.org/10.1093/nsr/nwad292

IF: 20.6

2023-01-01

National Science Review

Abstract:Formulating the methodology of machine learning by bilevel optimization techniques provides a new perspective to understand and solve automated machine learning problems.

What problem does this paper attempt to address?

Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

Kaiyi Ji

DOI: https://doi.org/10.48550/arXiv.2108.00330

IF: 5.414

2021-07-31

Machine Learning

Abstract:Bilevel optimization has become a powerful framework in various machine learning applications including meta-learning, hyperparameter optimization, and network architecture search. There are generally two classes of bilevel optimization formulations for machine learning: 1) problem-based bilevel optimization, whose inner-level problem is formulated as finding a minimizer of a given loss function; and 2) algorithm-based bilevel optimization, whose inner-level solution is an output of a fixed algorithm. For the first class, two popular types of gradient-based algorithms have been proposed for hypergradient estimation via approximate implicit differentiation (AID) and iterative differentiation (ITD). Algorithms for the second class include the popular model-agnostic meta-learning (MAML) and almost no inner loop (ANIL). However, the convergence rate and fundamental limitations of bilevel optimization algorithms have not been well explored. This thesis provides a comprehensive convergence rate analysis for bilevel algorithms in the aforementioned two classes. We further propose principled algorithm designs for bilevel optimization with higher efficiency and scalability. For the problem-based formulation, we provide a convergence rate analysis for AID- and ITD-based bilevel algorithms. We then develop acceleration bilevel algorithms, for which we provide shaper convergence analysis with relaxed assumptions. We also provide the first lower bounds for bilevel optimization, and establish the optimality by providing matching upper bounds under certain conditions. We finally propose new stochastic bilevel optimization algorithms with lower complexity and higher efficiency in practice. For the algorithm-based formulation, we develop a theoretical convergence for general multi-step MAML and ANIL, and characterize the impact of parameter selections and loss geometries on the their complexities.
Bilevel Optimization: Theory, Algorithms, Applications and a Bibliography

Stephan Dempe

DOI: https://doi.org/10.1007/978-3-030-52119-6_20

2020-01-01

Abstract:Bilevel optimization problems are hierarchical optimization problems where the feasible region of the so-called upper level problem is restricted by the graph of the solution set mapping of the lower level problem. Aim of this article is to collect a large number of references on this topic, to show the diversity of contributions and to support young colleagues who try to start research in this challenging and interesting field.
Functional Bilevel Optimization for Machine Learning

Ieva Petrulionyte,Julien Mairal,Michael Arbel

2024-06-13

Abstract:In this paper, we introduce a new functional point of view on bilevel optimization problems for machine learning, where the inner objective is minimized over a function space. These types of problems are most often solved by using methods developed in the parametric setting, where the inner objective is strongly convex with respect to the parameters of the prediction function. The functional point of view does not rely on this assumption and notably allows using over-parameterized neural networks as the inner prediction function. We propose scalable and efficient algorithms for the functional bilevel optimization problem and illustrate the benefits of our approach on instrumental regression and reinforcement learning tasks.

Machine Learning
Investigating Bi-Level Optimization for Learning and Vision From a Unified Perspective: A Survey and Beyond

Risheng Liu,Jiaxin Gao,Jin Zhang,Deyu Meng,Zhouchen Lin

DOI: https://doi.org/10.1109/TPAMI.2021.3132674

Abstract:Bi-Level Optimization (BLO) is originated from the area of economic game theory and then introduced into the optimization community. BLO is able to handle problems with a hierarchical structure, involving two levels of optimization tasks, where one task is nested inside the other. In machine learning and computer vision fields, despite the different motivations and mechanisms, a lot of complex problems, such as hyper-parameter optimization, multi-task and meta learning, neural architecture search, adversarial learning and deep reinforcement learning, actually all contain a series of closely related subproblms. In this paper, we first uniformly express these complex learning and vision problems from the perspective of BLO. Then we construct a best-response-based single-level reformulation and establish a unified algorithmic framework to understand and formulate mainstream gradient-based BLO methodologies, covering aspects ranging from fundamental automatic differentiation schemes to various accelerations, simplifications, extensions and their convergence and complexity properties. Last but not least, we discuss the potentials of our unified BLO framework for designing new algorithms and point out some promising directions for future research. A list of important papers discussed in this survey, corresponding codes, and additional resources on BLOs are publicly available at: https://github.com/vis-opt-group/BLO.
Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis

Jie Hao,Xiaochuan Gong,Mingrui Liu

2024-01-18

Abstract:Bilevel optimization is an important formulation for many machine learning problems. Current bilevel optimization algorithms assume that the gradient of the upper-level function is Lipschitz. However, recent studies reveal that certain neural networks such as recurrent neural networks (RNNs) and long-short-term memory networks (LSTMs) exhibit potential unbounded smoothness, rendering conventional bilevel optimization algorithms unsuitable. In this paper, we design a new bilevel optimization algorithm, namely BO-REP, to address this challenge. This algorithm updates the upper-level variable using normalized momentum and incorporates two novel techniques for updating the lower-level variable: \textit{initialization refinement} and \textit{periodic updates}. Specifically, once the upper-level variable is initialized, a subroutine is invoked to obtain a refined estimate of the corresponding optimal lower-level variable, and the lower-level variable is updated only after every specific period instead of each iteration. When the upper-level problem is nonconvex and unbounded smooth, and the lower-level problem is strongly convex, we prove that our algorithm requires $\widetilde{\mathcal{O}}(1/\epsilon^4)$ iterations to find an $\epsilon$-stationary point in the stochastic setting, where each iteration involves calling a stochastic gradient or Hessian-vector product oracle. Notably, this result matches the state-of-the-art complexity results under the bounded smoothness setting and without mean-squared smoothness of the stochastic gradient, up to logarithmic factors. Our proof relies on novel technical lemmas for the periodically updated lower-level variable, which are of independent interest. Our experiments on hyper-representation learning, hyperparameter optimization, and data hyper-cleaning for text classification tasks demonstrate the effectiveness of our proposed algorithm.

Machine Learning,Optimization and Control
Effective Bilevel Optimization via Minimax Reformulation

Xiaoyu Wang,Rui Pan,Renjie Pi,Jipeng Zhang

2024-11-04

Abstract:Bilevel optimization has found successful applications in various machine learning problems, including hyper-parameter optimization, data cleaning, and meta-learning. However, its huge computational cost presents a significant challenge for its utilization in large-scale problems. This challenge arises due to the nested structure of the bilevel formulation, where each hyper-gradient computation necessitates a costly inner optimization procedure. To address this issue, we propose a reformulation of bilevel optimization as a minimax problem, effectively decoupling the outer-inner dependency. Under mild conditions, we show these two problems are equivalent. Furthermore, we introduce a multi-stage gradient descent and ascent (GDA) algorithm to solve the resulting minimax problem with convergence guarantees. Extensive experimental results demonstrate that our method outperforms state-of-the-art bilevel methods while significantly reducing the computational cost.

Machine Learning,Optimization and Control
A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning

Ankur Sinha,Tanmay Khandait,Raja Mohanty

DOI: https://doi.org/10.48550/arXiv.2007.11022

IF: 5.414

2020-07-21

Machine Learning

Abstract:Hyperparameter tuning is an active area of research in machine learning, where the aim is to identify the optimal hyperparameters that provide the best performance on the validation set. Hyperparameter tuning is often achieved using naive techniques, such as random search and grid search. However, most of these methods seldom lead to an optimal set of hyperparameters and often get very expensive. In this paper, we propose a bilevel solution method for solving the hyperparameter optimization problem that does not suffer from the drawbacks of the earlier studies. The proposed method is general and can be easily applied to any class of machine learning algorithms. The idea is based on the approximation of the lower level optimal value function mapping, which is an important mapping in bilevel optimization and helps in reducing the bilevel problem to a single level constrained optimization task. The single-level constrained optimization problem is solved using the augmented Lagrangian method. We discuss the theory behind the proposed algorithm and perform extensive computational study on two datasets that confirm the efficiency of the proposed method. We perform a comparative study against grid search, random search and Bayesian optimization techniques that shows that the proposed algorithm is multiple times faster on problems with one or two hyperparameters. The computational gain is expected to be significantly higher as the number of hyperparameters increase. Corresponding to a given hyperparameter most of the techniques in the literature often assume a unique optimal parameter set that minimizes loss on the training set. Such an assumption is often violated by deep learning architectures and the proposed method does not require any such assumption.
Decentralized bilevel optimization

Xuxing Chen,Minhui Huang,Shiqian Ma

DOI: https://doi.org/10.1007/s11590-024-02101-4

IF: 1.5288

2024-03-28

Optimization Letters

Abstract:Bilevel optimization has been successfully applied to many important machine learning problems. Algorithms for solving bilevel optimization have been studied under various settings. In this paper, we study the nonconvex-strongly-convex bilevel optimization under a decentralized setting. We design decentralized algorithms for both deterministic and stochastic bilevel optimization problems. Moreover, we analyze the convergence rates of the proposed algorithms in difference scenarios including the case where data heterogeneity is observed across agents. Numerical experiments on both synthetic and real data demonstrate that the proposed methods are efficient.

mathematics, applied,operations research & management science
Bilevel Optimization based on Iterative Approximation of Multiple Mappings

Ankur Sinha,Zhichao Lu,Kalyanmoy Deb,Pekka Malo

DOI: https://doi.org/10.48550/arXiv.1702.03394

2017-05-06

Abstract:A large number of application problems involve two levels of optimization, where one optimization task is nested inside the other. These problems are known as bilevel optimization problems and have been studied by both classical optimization community and evolutionary optimization community. Most of the solution procedures proposed until now are either computationally very expensive or applicable to only small classes of bilevel optimization problems adhering to mathematically simplifying assumptions. In this paper, we propose an evolutionary optimization method that tries to reduce the computational expense by iteratively approximating two important mappings in bilevel optimization; namely, the lower level rational reaction mapping and the lower level optimal value function mapping. The algorithm has been tested on a large number of test problems and comparisons have been performed with other algorithms. The results show the performance gain to be quite significant. To the best knowledge of the authors, a combined theory-based and population-based solution procedure utilizing mappings has not been suggested yet for bilevel problems.

Optimization and Control
A Framework for Bilevel Optimization on Riemannian Manifolds

Andi Han,Bamdev Mishra,Pratik Jawanpuria,Akiko Takeda

2024-11-02

Abstract:Bilevel optimization has gained prominence in various applications. In this study, we introduce a framework for solving bilevel optimization problems, where the variables in both the lower and upper levels are constrained on Riemannian manifolds. We present several hypergradient estimation strategies on manifolds and analyze their estimation errors. Furthermore, we provide comprehensive convergence and complexity analyses for the proposed hypergradient descent algorithm on manifolds. We also extend our framework to encompass stochastic bilevel optimization and incorporate the use of general retraction. The efficacy of the proposed framework is demonstrated through several applications.

Optimization and Control,Machine Learning
A Primal-Dual Approach to Bilevel Optimization with Multiple Inner Minima

Daouda Sow,Kaiyi Ji,Ziwei Guan,Yingbin Liang

DOI: https://doi.org/10.48550/arXiv.2203.01123

2022-03-01

Optimization and Control

Abstract:Bilevel optimization has found extensive applications in modern machine learning problems such as hyperparameter optimization, neural architecture search, meta-learning, etc. While bilevel problems with a unique inner minimal point (e.g., where the inner function is strongly convex) are well understood, such a problem with multiple inner minimal points remains to be challenging and open. Existing algorithms designed for such a problem were applicable to restricted situations and do not come with a full guarantee of convergence. In this paper, we adopt a reformulation of bilevel optimization to constrained optimization, and solve the problem via a primal-dual bilevel optimization (PDBO) algorithm. PDBO not only addresses the multiple inner minima challenge, but also features fully first-order efficiency without involving second-order Hessian and Jacobian computations, as opposed to most existing gradient-based bilevel algorithms. We further characterize the convergence rate of PDBO, which serves as the first known non-asymptotic convergence guarantee for bilevel optimization with multiple inner minima. Our experiments demonstrate desired performance of the proposed approach.
Unlocking Global Optimality in Bilevel Optimization: A Pilot Study

Quan Xiao,Tianyi Chen

2024-08-29

Abstract:Bilevel optimization has witnessed a resurgence of interest, driven by its critical role in trustworthy and efficient machine learning applications. Recent research has focused on proposing efficient methods with provable convergence guarantees. However, while many prior works have established convergence to stationary points or local minima, obtaining the global optimum of bilevel optimization remains an important yet open problem. The difficulty lies in the fact that unlike many prior non-convex single-level problems, this bilevel problem does not admit a ``benign" landscape, and may indeed have multiple spurious local solutions. Nevertheless, attaining the global optimality is indispensable for ensuring reliability, safety, and cost-effectiveness, particularly in high-stakes engineering applications that rely on bilevel optimization. In this paper, we first explore the challenges of establishing a global convergence theory for bilevel optimization, and present two sufficient conditions for global convergence. We provide algorithm-specific proofs to rigorously substantiate these sufficient conditions along the optimization trajectory, focusing on two specific bilevel learning scenarios: representation learning and data hypercleaning (a.k.a. reweighting). Experiments corroborate the theoretical findings, demonstrating convergence to global minimum in both cases.

Optimization and Control,Machine Learning
A Primal-Dual-Assisted Penalty Approach to Bilevel Optimization with Coupled Constraints

Liuyuan Jiang,Quan Xiao,Victor M. Tenorio,Fernando Real-Rojas,Antonio G. Marques,Tianyi Chen

2024-08-26

Abstract:Interest in bilevel optimization has grown in recent years, partially due to its applications to tackle challenging machine-learning problems. Several exciting recent works have been centered around developing efficient gradient-based algorithms that can solve bilevel optimization problems with provable guarantees. However, the existing literature mainly focuses on bilevel problems either without constraints, or featuring only simple constraints that do not couple variables across the upper and lower levels, excluding a range of complex applications. Our paper studies this challenging but less explored scenario and develops a (fully) first-order algorithm, which we term BLOCC, to tackle BiLevel Optimization problems with Coupled Constraints. We establish rigorous convergence theory for the proposed algorithm and demonstrate its effectiveness on two well-known real-world applications - hyperparameter selection in support vector machine (SVM) and infrastructure planning in transportation networks using the real data from the city of Seville.

Optimization and Control,Machine Learning
An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learning

Yihua Zhang,Prashant Khanduri,Ioannis Tsaknakis,Yuguang Yao,Mingyi Hong,Sijia Liu

2023-12-21

Abstract:Recently, bi-level optimization (BLO) has taken center stage in some very exciting developments in the area of signal processing (SP) and machine learning (ML). Roughly speaking, BLO is a classical optimization problem that involves two levels of hierarchy (i.e., upper and lower levels), wherein obtaining the solution to the upper-level problem requires solving the lower-level one. BLO has become popular largely because it is powerful in modeling problems in SP and ML, among others, that involve optimizing nested objective functions. Prominent applications of BLO range from resource allocation for wireless systems to adversarial machine learning. In this work, we focus on a class of tractable BLO problems that often appear in SP and ML applications. We provide an overview of some basic concepts of this class of BLO problems, such as their optimality conditions, standard algorithms (including their optimization principles and practical implementations), as well as how they can be leveraged to obtain state-of-the-art results for a number of key SP and ML applications. Further, we discuss some recent advances in BLO theory, its implications for applications, and point out some limitations of the state-of-the-art that require significant future research efforts. Overall, we hope that this article can serve to accelerate the adoption of BLO as a generic tool to model, analyze, and innovate on a wide array of emerging SP and ML applications.

Machine Learning,Optimization and Control
Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF

Han Shen,Zhuoran Yang,Tianyi Chen

2024-06-01

Abstract:Bilevel optimization has been recently applied to many machine learning tasks. However, their applications have been restricted to the supervised learning setting, where static objective functions with benign structures are considered. But bilevel problems such as incentive design, inverse reinforcement learning (RL), and RL from human feedback (RLHF) are often modeled as dynamic objective functions that go beyond the simple static objective structures, which pose significant challenges of using existing bilevel solutions. To tackle this new class of bilevel problems, we introduce the first principled algorithmic framework for solving bilevel RL problems through the lens of penalty formulation. We provide theoretical studies of the problem landscape and its penalty-based (policy) gradient algorithms. We demonstrate the effectiveness of our algorithms via simulations in the Stackelberg Markov game, RL from human feedback and incentive design.

Machine Learning,Optimization and Control
A Bilevel Optimization Framework for Imbalanced Data Classification

Karen Medlin,Sven Leyffer,Krishnan Raghavan

2024-10-15

Abstract:Data rebalancing techniques, including oversampling and undersampling, are a common approach to addressing the challenges of imbalanced data. To tackle unresolved problems related to both oversampling and undersampling, we propose a new undersampling approach that: (i) avoids the pitfalls of noise and overlap caused by synthetic data and (ii) avoids the pitfall of under-fitting caused by random undersampling. Instead of undersampling majority data randomly, our method undersamples datapoints based on their ability to improve model loss. Using improved model loss as a proxy measurement for classification performance, our technique assesses a datapoint's impact on loss and rejects those unable to improve it. In so doing, our approach rejects majority datapoints redundant to datapoints already accepted and, thereby, finds an optimal subset of majority training data for classification. The accept/reject component of our algorithm is motivated by a bilevel optimization problem uniquely formulated to identify the optimal training set we seek. Experimental results show our proposed technique with F1 scores up to 10% higher than state-of-the-art methods.

Machine Learning,Optimization and Control
Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysis

Yifan Yang,Hao Ban,Minhui Huang,Shiqian Ma,Kaiyi Ji

2024-10-09

Abstract:Bilevel optimization has recently attracted considerable attention due to its abundant applications in machine learning problems. However, existing methods rely on prior knowledge of problem parameters to determine stepsizes, resulting in significant effort in tuning stepsizes when these parameters are unknown. In this paper, we propose two novel tuning-free algorithms, D-TFBO and S-TFBO. D-TFBO employs a double-loop structure with stepsizes adaptively adjusted by the "inverse of cumulative gradient norms" strategy. S-TFBO features a simpler fully single-loop structure that updates three variables simultaneously with a theory-motivated joint design of adaptive stepsizes for all variables. We provide a comprehensive convergence analysis for both algorithms and show that D-TFBO and S-TFBO respectively require $O(\frac{1}{\epsilon})$ and $O(\frac{1}{\epsilon}\log^4(\frac{1}{\epsilon}))$ iterations to find an $\epsilon$-accurate stationary point, (nearly) matching their well-tuned counterparts using the information of problem parameters. Experiments on various problems show that our methods achieve performance comparable to existing well-tuned approaches, while being more robust to the selection of initial stepsizes. To the best of our knowledge, our methods are the first to completely eliminate the need for stepsize tuning, while achieving theoretical guarantees.

Machine Learning
A Single-Loop Algorithm for Decentralized Bilevel Optimization

Youran Dong,Shiqian Ma,Junfeng Yang,Chao Yin

2024-04-23

Abstract:Bilevel optimization has gained significant attention in recent years due to its broad applications in machine learning. This paper focuses on bilevel optimization in decentralized networks and proposes a novel single-loop algorithm for solving decentralized bilevel optimization with a strongly convex lower-level problem. Our approach is a fully single-loop method that approximates the hypergradient using only two matrix-vector multiplications per iteration. Importantly, our algorithm does not require any gradient heterogeneity assumption, distinguishing it from existing methods for decentralized bilevel optimization and federated bilevel optimization. Our analysis demonstrates that the proposed algorithm achieves the best-known convergence rate for bilevel optimization algorithms. We also present experimental results on hyperparameter optimization problems using both synthetic and MNIST datasets, which demonstrate the efficiency of our proposed algorithm.

Optimization and Control,Distributed, Parallel, and Cluster Computing,Machine Learning
An Enhanced Branch-and-bound Algorithm for Bilevel Integer Linear Programming.

Shaonan Liu,Mingzheng Wang,Nan Kong,Xiangpei Hu

DOI: https://doi.org/10.1016/j.ejor.2020.10.002

IF: 6.4

2020-01-01

European Journal of Operational Research

Abstract:Bilevel integer linear programming (BILP) problems have been studied for decades. Many exact algorithms have been proposed in recent years for small- or medium-sized instances. However, few of these algorithms were shown to be efficient on large-sized instances. In this paper, we present an enhanced branch-and-bound algorithm for a class of BILP problems, which can discard a subspace from the search space in each iteration larger than that in a benchmark branch-and-bound algorithm. The corresponding enhanced branching rule can efficiently slow down the creation of new node problems so as to significantly reduce the computation time. Our scheme may be suboptimal if the lower-level problem is not unique optimal as the enhanced branching rule may discard bilevel feasible solutions that may turn out to be optimal to the bilevel programming. We present computational studies to evaluate the algorithm speedup and solution quality of our algorithm, compared with state-of-the-art algorithms from the literature on a large testbed of general BILP instances, some of which are still unsolved. The computational results show that our enhanced branching rule can achieve significant speedup on the benchmark branching rule with satisfying solution quality. In particular, our algorithm shows superior performance on large-sized BILP instances with a relatively complex lower-level problem. (C) 2020 Elsevier B.V. All rights reserved.
Riemannian Bilevel Optimization

Jiaxiang Li,Shiqian Ma

2024-02-08

Abstract:In this work, we consider the bilevel optimization problem on Riemannian manifolds. We inspect the calculation of the hypergradient of such problems on general manifolds and thus enable the utilization of gradient-based algorithms to solve such problems. The calculation of the hypergradient requires utilizing the notion of Riemannian cross-derivative and we inspect the properties and the numerical calculations of Riemannian cross-derivatives. Algorithms in both deterministic and stochastic settings, named respectively RieBO and RieSBO, are proposed that include the existing Euclidean bilevel optimization algorithms as special cases. Numerical experiments on robust optimization on Riemannian manifolds are presented to show the applicability and efficiency of the proposed methods.

Optimization and Control

Bilevel Optimization for Automated Machine Learning: a New Perspective on Framework and Algorithm

Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

Bilevel Optimization: Theory, Algorithms, Applications and a Bibliography

Functional Bilevel Optimization for Machine Learning

Investigating Bi-Level Optimization for Learning and Vision From a Unified Perspective: A Survey and Beyond

Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis

Effective Bilevel Optimization via Minimax Reformulation

A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning

Decentralized bilevel optimization

Bilevel Optimization based on Iterative Approximation of Multiple Mappings

A Framework for Bilevel Optimization on Riemannian Manifolds

A Primal-Dual Approach to Bilevel Optimization with Multiple Inner Minima

Unlocking Global Optimality in Bilevel Optimization: A Pilot Study

A Primal-Dual-Assisted Penalty Approach to Bilevel Optimization with Coupled Constraints

An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learning

Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF

A Bilevel Optimization Framework for Imbalanced Data Classification

Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysis

A Single-Loop Algorithm for Decentralized Bilevel Optimization

An Enhanced Branch-and-bound Algorithm for Bilevel Integer Linear Programming.

Riemannian Bilevel Optimization