Abstract:Model-based derivative-free optimization (DFO) methods are an important class of DFO methods that are known to struggle with solving high-dimensional optimization problems. Recent research has shown that incorporating random subspaces into model-based DFO methods has the potential to improve their performance on high-dimensional problems. However, most of the current theoretical and practical results are based on linear approximation models due to the complexity of quadratic approximation models. This paper proposes a random subspace trust-region algorithm based on quadratic approximations. Unlike most of its precursors, this algorithm does not require any special form of objective function. We study the geometry of sample sets, the error bounds for approximations, and the quality of subspaces. In particular, we provide a technique to construct $Q$-fully quadratic models, which is easy to analyze and implement. We present an almost-sure global convergence result of our algorithm and give an upper bound on the expected number of iterations to find a sufficiently small gradient. We also develop numerical experiments to compare the performance of our algorithm using both linear and quadratic approximation models. The numerical results demonstrate the strengths and weaknesses of using quadratic approximations.

A High-Order Recursive Quadratic Algorithm for Linear-in-the-Parameter Models

A robust high-order recursive quadratic algorithm for linear-in-the-parameters models

A high-order recursive quadratic learning algorithm

Recursive Least Squares Support Vector Regression

Modified recursive least squares algorithm with variable parameters and resetting for time-varying system

Improve recursive identification using multi-iteration

Recursive Least-Squares Method with Membership Functions

Nonlinear Robust Modeling Base on Least Trimmed Squares Regression

Recursive Variable Projection Algorithm for a Class of Separable Nonlinear Models

Two-dimensional recursive least squares identification based on local polynomial modeling for batch processes.

Sequential Quadratic Programming Method for Nonlinear Least Squares Estimation and Its Application

Recursive Least Squares for Censored Regression

Quadratic Message Passing for Generalized Quadratic Equations Model

Embedded Point Iteration Based Recursive Algorithm for Online Identification of Nonlinear Regression Models

Robust Kernel Recursive Adaptive Filtering Algorithms Based on M-estimate

Solution to the Rational Function Model Based on the Levenberg-Marquardt Algorithm

An Accelerated Proximal Gradient-Based Algorithm for Quadratic Programming

Two-dimensional Incremental Recursive Least Squares Identification for Batch Processes

A Tutorial on Linear Least Square Estimation

$Q$-fully Quadratic Modeling and its Application in a Random Subspace Derivative-free Method

Another simple recurrent neural network for quadratic and linear programming