Abstract:Derivative-free optimization (DFO) consists in finding the best value of an objective function without relying on derivatives. To tackle such problems, one may build approximate derivatives, using for instance finite-difference estimates. One may also design algorithmic strategies that perform space exploration and seek improvement over the current point. The first type of strategy often provides good performance on smooth problems but at the expense of more function evaluations. The second type is cheaper and typically handles non-smoothness or noise in the objective better. Recently, full-low evaluation methods have been proposed as a hybrid class of DFO algorithms that combine both strategies, respectively denoted as Full-Eval and Low-Eval. In the unconstrained case, these methods showed promising numerical performance. In this paper, we extend the full-low evaluation framework to bound and linearly constrained derivative-free optimization. We derive convergence results for an instance of this framework, that combines finite-difference quasi-Newton steps with probabilistic direct-search steps. The former are projected onto the feasible set, while the latter are defined within tangent cones identified by nearby active constraints. We illustrate the practical performance of our instance on standard linearly constrained problems, that we adapt to introduce noisy evaluations as well as non-smoothness. In all cases, our method performs favorably compared to algorithms that rely solely on Full-eval or Low-eval iterations.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the Derivative - Free Optimization (DFO) problem under boundary and linear constraints. Specifically, the author proposes an extended "Full - Low Evaluation" framework to handle the DFO problem with boundary and linear constraints. This method combines two different types of steps: 1. **Full - Eval steps**: These steps approximate the gradient by finite differences and use projection techniques to ensure the feasibility of the solution. These steps usually perform well when the objective function is smooth, but they require more function evaluations and are therefore more costly. 2. **Low - Eval steps**: These steps are based on probabilistic direct search. In each iteration, a small number of directions are selected for searching to find a feasible descent direction. These steps are more robust when the objective function is noisy or non - smooth, and the computational cost is lower. The main contribution of the paper is to combine these two strategies to form a hybrid algorithm framework that can maintain high performance and effectively handle non - smooth and noisy problems when dealing with the DFO problem with boundary and linear constraints. ### Specific problem description The optimization problem that the paper focuses on is a minimization problem with linear constraints, in the following form: \[ \min_{x \in \mathbb{R}^n} f(x) \quad \text{s.t.} \quad Ax = b, \quad \ell \leq A_I x \leq u, \] where \( f: \mathbb{R}^n \to \mathbb{R} \), \( A \in \mathbb{R}^{m \times n} \), \( A_I \in \mathbb{R}^{m_I \times n} \), \( b \in \mathbb{R}^m \), \( (\ell, u) \in \bar{\mathbb{R}}^{m_I} \times \bar{\mathbb{R}}^{m_I} \), and \( \bar{\mathbb{R}}=\mathbb{R} \cup \{-\infty, \infty\} \). ### Optimization method The optimization method proposed in the paper consists of two main parts: 1. **Full - Eval steps**: - Calculate the gradient \( g_k \) approximated by finite differences. - Use the projection technique \( P_F \) to project the search direction \( p_k=-g_k \) onto the feasible set to obtain \( \bar{x}_k = P_F[x_k + p_k] \). - Perform a line search to find a step size \( \beta_k \) that satisfies the sufficient descent condition. 2. **Low - Eval steps**: - Generate a set of feasible search directions \( D_k \). - Search along these directions to find a feasible point \( x_k+\alpha_k d_k \) that reduces the value of the objective function. ### Convergence analysis The paper also provides the convergence analysis of the algorithm, including the convergence rates in smooth and non - smooth cases. For the smooth case, it is proved that if the Full - Eval steps generate an infinite sequence of iteration points, then \( \|q(x_k)\| \) converges to zero at a rate of \( 1/\sqrt{k} \). For the non - smooth case, by using the sufficient descent property of the Low - Eval steps, it is proved that the algorithm parameter \( \alpha_k \) tends to zero, thus ensuring the global convergence of the algorithm. ### Experimental results The paper conducts experiments on standard linear - constraint problems to verify the effectiveness of the proposed method. The experimental results show that this method outperforms existing algorithms that rely only on Full - Eval or Low - Eval steps when dealing with both smooth and non - smooth problems. In conclusion, by combining Full - Eval and Low - Eval steps, this paper proposes a new Derivative - Free Optimization framework, which effectively solves the boundary and linear - constraint problems.

Full-Low Evaluation Methods For Bound and Linearly Constrained Derivative-Free Optimization

Full-low evaluation methods for derivative-free optimization

A Relative Feasibility Degree Based Approach for Constrained Optimization Problems.

On high-order multilevel optimization strategies

Linear Convergence of the Derivative-Free Proximal Bundle Method on Convex Nonsmooth Functions, with Application to the Derivative-Free -Algorithm

Black-box Optimization Algorithms for Regularized Least-squares Problems

Derivative-Free Optimization via Finite Difference Approximation: An Experimental Study

Improved Algorithms Based on Trust Region Framework for Solving Unconstrained Derivative Free Optimization Problems

Scalable Subspace Methods for Derivative-Free Nonlinear Least-Squares Optimization

An inertial hybrid DFPM-based algorithm for constrained nonlinear equations with applications

Curvature-Aware Derivative-Free Optimization

Global optimization advances in Mixed-Integer Nonlinear Programming, MINLP, and Constrained Derivative-Free Optimization, CDFO

Unifying nonlinearly constrained nonconvex optimization

Deterministic Langevin Unconstrained Optimization with Normalizing Flows

Revisiting Theoretical Guarantees of Direct-Search Methods

A new unified framework for designing convex optimization methods with prescribed theoretical convergence estimates: A numerical analysis approach

Analysis of Optimization Algorithms via Sum-of-Squares

Learning to optimize with convergence guarantees using nonlinear system theory

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

Extending oscars-ii to generally constrained global optimization