Abstract:Previous chapter Next chapter Advances in Design and Control Control Perspectives on Numerical Algorithms and Matrix Problems4. Neural-Gradient Dynamical Systems for Linear and Quadratic Programming Problemspp.127 - 177Chapter DOI:https://doi.org/10.1137/1.9780898718669.ch4PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAboutExcerpt At first glance, Liapunov's works were not related to optimization. However, this is not exactly the case. Liapunov developed stability theory for ordinary differential equations; in its simplest form it states that a solution x (t) of the equation x˙ = f (x) is stable if there exists a function V (x) (the Liapunov function) such that 〈∇ V (x), f (x)〉 < 0. We can take a reverse point of view: the differential equation above is a continuous-time method for minimizing V (x). Thus the method provides a systematic tool for validation of the convergence of numerical methods of optimization. —B. T. Polyak [Pol02] This chapter develops the point of view advocated in the quote above, adding the ingredients of control parameters and control Liapunov functions. From the perspective of Chapter 2, continuous algorithms developed in this chapter for finding extrema of functions are given in the form of ODEs whose equilibria are identical with the extrema being sought. In fact, linear and quadratic programming problems are solved using the class of ODEs called gradient dynamical systems (GDSs). These optimization problems are transformed into unconstrained optimization problems using exact penalty functions, which are then solved using a gradient descent method, i.e., a GDS. The key idea is to adjust the penalty parameters, interpreted as control parameters, in the resulting GDS, using a control Liapunov function (CLF) approach. Since the system being studied is a GDS, the energy or potential function whose gradient defines the right-hand side (of the GDS) is a natural Liapunov function. The fact that these GDFSs can be represented and implemented as a class of recurrent neural networks explains the neologism neural-gradient dynamical systems that occurs in the chapter title, essentially to call the reader's attention to the equivalence between a GDS approach to optimization and the recurrent neural network approach. Anovelty that arises in the approach presented in this chapter is that nonsmooth CLFs have to be used since exact penalty functions are nonsmooth (i.e., have finitely many points of nondifferentiability). This technicality is handled using a generalized Persidskii-type result, presented in Chapter 1, that allows the treatment of a class of differential equations with discontinuous right-hand sides. Previous chapter Next chapter RelatedDetails Published:2006ISBN:978-0-89871-602-3eISBN:978-0-89871-866-9 https://doi.org/10.1137/1.9780898718669Book Series Name:Advances in Design and ControlBook Code:DC10Book Pages:xxv + 265Key words:Control theory, system theory, iterative methods, numerical methods for ODEs, Liapunov functions, gradient dynamical systems

The Controlled Conjugate Gradient Type Trajectory-Following Neural Net for Minimization of Nonconvex Functions.

Design of Second Order Neural Networks As Dynamical Control Systems That Aim to Minimize Nonconvex Scalar Functions

Unified Control Liapunov Function Based Design of Neural Networks That Aim at Global Minimization of Nonconvex Functions

Comparative Study of the CG and HBF ODEs Used in the Global Minimization of Nonconvex Functions

Successive Convexification for Trajectory Optimization with Continuous-Time Constraint Satisfaction

Generalized Levenberg-Marquardt Neural Nets for Minimization of Quasiconvex Scalar Functions

Trajectory-Oriented Control Using Gradient Descent: An Unconventional Approach

Generalized Gradient Projection Neural Networks for Nonsmooth Optimization Problems

A discrete-time neural network for optimization problems with hybrid constraints.

Accelerated Gradient Approach For Deep Neural Network-Based Adaptive Control of Unknown Nonlinear Systems

Subgradient-based Feedback Neural Networks for Non-Differentiable Convex Optimization Problems

Closing the gap: Optimizing Guidance and Control Networks through Neural ODEs

Neural-Gradient Dynamical Systems for Linear and Quadratic Programming Problems

Online Trajectory Optimization Using Inexact Gradient Feedback for Time-Varying Environments

Control Liapunov function design of neural networks that solve convex optimization and variational inequality problems

Lifelong Learning-Based Optimal Trajectory Tracking Control of Constrained Nonlinear Affine Systems Using Deep Neural Networks

A Numerical Study of Basins of Attraction of Zero-Finding Neural Nets Designed Using Control Theory

Variational formulations of ODE-Net as a mean-field optimal control problem and existence results

Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima

Intelligent acceleration adaptive control of linear $2\times2$ hyperbolic PDE systems

Limiting Behaviors of Nonconvex-Nonconcave Minimax Optimization via Continuous-Time Systems