Abstract:When faced with multiple minima of an "inner-level" convex optimization problem, the convex bilevel optimization problem selects an optimal solution which also minimizes an auxiliary "outer-level" convex objective of interest. Bilevel optimization requires a different approach compared to single-level optimization problems since the set of minimizers for the inner-level objective is not given explicitly. In this paper, we propose a new projection-free method for convex bilevel optimization which require only a linear optimization oracle over the base domain. We establish $O(t^{-1/2})$ convergence rate guarantees for our method in terms of both inner- and outer-level objectives, and demonstrate how additional assumptions such as quadratic growth and strong convexity result in accelerated rates of up to $O(t^{-1})$ and $O(t^{-2/3})$ for inner- and outer-levels respectively. Lastly, we conduct a numerical study to demonstrate the performance of our method.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to find an optimal solution in convex bilevel optimization problems while minimizing the auxiliary outer objective function. Specifically, when faced with multiple minima of the inner optimization problem, the convex bilevel optimization problem selects an optimal solution that not only satisfies the minimization of the inner objective function but also minimizes the outer objective function. Such problems are very common in practical applications, for example, finding the minimum - norm solution in underdetermined least - squares problems, or being applied in dictionary learning, fair classification and ill - posed inverse problems.
### Background and Problem Description of the Paper
Bilevel optimization problems involve two - level objective functions: an inner objective function \(g(z)\) and an outer objective function \(f(x)\). When \(g(z)\) is convex on a certain domain \(X\), its optimal solution set \(X_{opt}\) is also convex. If \(f(x)\) is also convex, then the entire bilevel optimization problem can be formulated as a convex optimization problem:
\[ \min_{x\in X}f(x)\quad\text{s.t.}\quad x\in X_{opt}:=\arg\min_{z\in X}g(z). \]
### Main Challenges
1. **Explicit Representation of the Optimal Solution Set**: Usually, we do not have an explicit representation of \(X_{opt}\), which makes it impossible to directly use common operations such as projection or linear optimization.
2. **Slater Condition Not Satisfied**: Due to the constraint of \(g(x)\leq g_{opt}\), problem (3) does not satisfy the Slater condition, which means that its dual problem may be unsolvable.
### Research Methods
To solve the above problems, the paper proposes a new projection - free conditional gradient method, which only needs to perform linear optimization on the base domain \(X\). The specific steps are as follows:
1. **Iterative Regularized Conditional Gradient Method (IR - CG)**: By introducing a regularization parameter \(\sigma_t\), the original problem is transformed into a regularized problem, and the conditional gradient update is used to gradually approximate the optimal solution.
2. **Convergence Rate Analysis**: Under different assumptions, the convergence rate of the algorithm is proved. For example, in general cases, the convergence rates of the inner and outer objective functions are \(O(t^{-p})\) and \(O(t^{-(1 - p)})\) respectively, where \(p\in(0,1)\).
3. **Accelerated Convergence**: Under additional assumptions (such as quadratic growth and strong convexity), the convergence rate of the algorithm can be further accelerated to \(O(t^{-\min\{1,2p\}})\) and \(O(t^{-\min\{p,1 - p\}})\).
### Experimental Results
The paper also conducts numerical experiments to compare the performance of the proposed method with existing methods (such as IR - PG, Bi - SG and CG - BiO) in matrix completion problems. The experimental results show that the IR - CG method is superior to the existing methods in performance, verifying the effectiveness of the theoretical analysis.
### Summary
This paper proposes a new projection - free conditional gradient method for solving convex bilevel optimization problems. By introducing a regularization parameter and conditional gradient update, this method can effectively solve bilevel optimization problems without explicitly representing the optimal solution set and has good convergence performance.