Abstract:This is a tutorial and survey paper on Karush-Kuhn-Tucker (KKT) conditions, first-order and second-order numerical optimization, and distributed optimization. After a brief review of history of optimization, we start with some preliminaries on properties of sets, norms, functions, and concepts of optimization. Then, we introduce the optimization problem, standard optimization problems (including linear programming, quadratic programming, and semidefinite programming), and convex problems. We also introduce some techniques such as eliminating inequality, equality, and set constraints, adding slack variables, and epigraph form. We introduce Lagrangian function, dual variables, KKT conditions (including primal feasibility, dual feasibility, weak and strong duality, complementary slackness, and stationarity condition), and solving optimization by method of Lagrange multipliers. Then, we cover first-order optimization including gradient descent, line-search, convergence of gradient methods, momentum, steepest descent, and backpropagation. Other first-order methods are explained, such as accelerated gradient method, stochastic gradient descent, mini-batch gradient descent, stochastic average gradient, stochastic variance reduced gradient, AdaGrad, RMSProp, and Adam optimizer, proximal methods (including proximal mapping, proximal point algorithm, and proximal gradient method), and constrained gradient methods (including projected gradient method, projection onto convex sets, and Frank-Wolfe method). We also cover non-smooth and $\ell_1$ optimization methods including lasso regularization, convex conjugate, Huber function, soft-thresholding, coordinate descent, and subgradient methods. Then, we explain second-order methods including Newton's method for unconstrained, equality constrained, and inequality constrained problems....

Constrained optimization in simulation: efficient global optimization and Karush-Kuhn-Tucker conditions

An Efficient Kriging-Based Constrained Optimization Algorithm by Global and Local Sampling in Feasible Region

A simulation-based approach for solving optimisation problems with ODE-type steady state constraints

A Novel Gaussian Process Surrogate Model with Expected Prediction Error for Optimization under Constraints

A kriging-assisted bi-objective constrained global optimization algorithm for expensive constrained optimization problems

Global Optimization: A Machine Learning Approach

Constrained efficient global multidisciplinary design optimization using adaptive disciplinary surrogate enrichment

Extending oscars-ii to generally constrained global optimization

Bayesian Optimisation for Constrained Problems

Parallelized multiobjective efficient global optimization algorithm and its applications

SMGO-$Δ$: Balancing Caution and Reward in Global Optimization with Black-Box Constraints

Surrogate-Based Simulation Optimization

KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey

A global optimization strategy based on the Kriging surrogate model and parallel computing

Surrogate-based optimization with clustering-based space exploration for expensive multimodal problems

Multi-objective efficient global optimization of expensive simulation-based problem in presence of simulation failures

On Enhanced KKT Optimality Conditions for Smooth Nonlinear Optimization

An efficient global optimization algorithm for expensive constrained black-box problems by reducing candidate infilling region

Kriging-assisted Discrete Global Optimization (KDGO) for black-box problems with costly objective and constraints

Global Optimization of Gaussian processes