Abstract:We present a new feasible proximal gradient method for constrained optimization where both the objective and constraint functions are given by the summation of a smooth, possibly nonconvex function and a convex simple function. The algorithm converts the original problem into a sequence of convex subproblems. Formulating those subproblems requires the evaluation of at most one gradient value of the original objective and constraint functions. Either exact or approximate subproblem solutions can be computed efficiently in many cases. An important feature of the algorithm is the constraint level parameter. By carefully increasing this level for each subproblem, we provide a simple solution to overcome the challenge of bounding the Lagrangian multipliers and show that the algorithm follows a strictly feasible solution path till convergence to the stationary point. We develop a simple, proximal gradient descent type analysis, showing that the complexity bound of this new algorithm is comparable to gradient descent for the unconstrained setting, which is new in the literature. Exploiting this new design and analysis technique, we extend our algorithms to some more challenging constrained optimization problems where 1) the objective is a stochastic or finite-sum function, and 2) structured nonsmooth functions replace smooth components of both objective and constraint functions. Complexity results for these problems also seem to be new in the literature. Finally, our method can also be applied to convex function-constrained problems where we show complexities similar to the proximal gradient method.
What problem does this paper attempt to address?
This paper aims to address several key challenges in function - constrained optimization problems. Specifically, the paper focuses on the following form of constrained optimization problems:
\[
\min_{x \in \mathbb{R}^d} \psi_0(x) = f_0(x) + \chi_0(x) \\
\text{s.t. } \psi_i(x) = f_i(x) + \chi_i(x) \leq \eta_i, \quad i = 1, \ldots, m.
\]
Here, \(\psi_i(x)\) is a composite function composed of a smooth but possibly non - convex function \(f_i(x)\) and a simple convex function \(\chi_i(x)\). Both the objective function and the constraint functions have this structure.
### Main Problems
1. **Efficient Solution**: Existing methods usually need to solve convex sub - problems exactly when dealing with large - scale optimization problems, which is often unrealistic in practical applications. The paper proposes a new Feasible Proximal Gradient method (LCPG), which can effectively handle these sub - problems even if they can only be approximately solved.
2. **Constraint Level Parameter**: By introducing a constraint level parameter and gradually increasing this parameter in each sub - problem, the paper provides a simple method to overcome the bound value problem of Lagrange multipliers. This method ensures that the algorithm always follows a strictly feasible solution path before converging to a stable point.
3. **Complexity Analysis**: The paper conducts a complexity analysis of the new algorithm and proves that its complexity is comparable to that of the gradient descent method in unconstrained optimization. This is the first reported result in the literature.
4. **Extended Applications**: The paper further extends the algorithm to more complex constrained optimization problems, including:
- The case where the objective function is a stochastic or finite - sum function.
- The objective and constraint functions contain structured non - smooth parts.
### Main Contributions
- **New Algorithm**: Proposes the Level Constrained Proximal Gradient (LCPG) method, which solves the original problem by transforming it into a series of convex sub - problems.
- **Complexity Analysis**: Proves that the complexity of the LCPG method to find the first - order \(\epsilon\)-KKT point in the case of non - convex composite functions is \(O(1/\epsilon)\), and each sub - problem only requires one function value and gradient calculation.
- **Extended Applications**: Extends the LCPG method to stochastic optimization, finite - sum optimization, and non - smooth non - convex function - constrained optimization problems, and provides the corresponding complexity results.
- **Theoretical Guarantee**: Under the condition of satisfying the Mangasarian - Fromovitz Constraint Qualification (MFCQ), proves the asymptotic convergence of the LCPG method.
In conclusion, by proposing the LCPG method, this paper not only solves the efficiency and feasibility problems of existing methods in dealing with large - scale optimization problems, but also provides a new theoretical analysis framework, laying the foundation for future research.