Methods of Nonconvex Optimization

V. S. Mikhalevich,A. M. Gupal,V. I. Norkin

2024-06-15

Abstract:This book is devoted to finite-dimensional problems of non-convex non-smooth optimization and numerical methods for their solution. The problem of nonconvexity is studied in the book on two main models of nonconvex dependencies: these are the so-called generalized differentiable functions and locally Lipschitz functions. Non-smooth functions naturally arise in various applications. In addition, they often appear in the theory of extremal problems itself due to the operations of taking the maximum and minimum, decomposition techniques, exact non-smooth penalties, and duality. The considered models of nonconvexity are quite general and cover the majority of practically important optimization problems; they clearly show all the difficulties of non-convex optimization. The method of studying the generalized differentiable functions is that for these functions a generalization of the concept of gradient is introduced, a calculus is constructed, and various properties of nonconvex problems are studied in terms of generalized gradients. As for numerical methods, it is possible to extend the theory and algorithms of subgradient descent of convex optimization to problems with generalized differentiable functions. Methods for solving Lipschitz problems are characterized by the fact that the original functions are approximated by smoothed ones and iterative minimization procedures are applied to them. With this approach, it is possible to approximate the gradients of smoothed functions by stochastic finite differences and thus to construct methods without calculating gradients. A similar approach can be justified in generalized differentiable and Lipschitz stochastic programming. In these cases, various generalizations of the classical stochastic approximation and stochastic quasi-gradient method are obtained for solving constrained nonconvex nonsmooth stochastic programming problems.

Optimization and Control

What problem does this paper attempt to address?

This paper attempts to address non-convex and non-smooth optimization problems. Specifically, it explores finite-dimensional non-convex and non-smooth optimization problems and their corresponding numerical solutions. Since non-differentiable functions and non-convex regions are often encountered in application problems, developing optimization methods to solve such problems is a significant challenge. The book covers the basic theory of non-convex analysis, smoothing techniques for locally Lipschitz functions, necessary optimality conditions, and more. It also proposes several numerical methods, such as the finite difference method, stochastic finite difference direction method, and generalized gradient descent method. These methods are suitable for solving non-convex and non-smooth optimization problems and are computationally efficient and practical.

Additionally, the book discusses how to handle multi-extremum problems and potential stochastic properties, proposing the combination of algorithms and other global search strategies to overcome multi-extremum issues. By introducing the concept of generalized differentiable functions, it becomes possible to construct numerical methods using pseudo-gradients, thereby simplifying the solution process of non-convex optimization problems. In summary, the book aims to provide a theoretical foundation and practical tools for solving complex non-convex and non-smooth optimization problems in practical applications.