Abstract:Previous chapter Next chapter Advances in Design and Control Control Perspectives on Numerical Algorithms and Matrix Problems4. Neural-Gradient Dynamical Systems for Linear and Quadratic Programming Problemspp.127 - 177Chapter DOI:https://doi.org/10.1137/1.9780898718669.ch4PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAboutExcerpt At first glance, Liapunov's works were not related to optimization. However, this is not exactly the case. Liapunov developed stability theory for ordinary differential equations; in its simplest form it states that a solution x (t) of the equation x˙ = f (x) is stable if there exists a function V (x) (the Liapunov function) such that 〈∇ V (x), f (x)〉 < 0. We can take a reverse point of view: the differential equation above is a continuous-time method for minimizing V (x). Thus the method provides a systematic tool for validation of the convergence of numerical methods of optimization. —B. T. Polyak [Pol02] This chapter develops the point of view advocated in the quote above, adding the ingredients of control parameters and control Liapunov functions. From the perspective of Chapter 2, continuous algorithms developed in this chapter for finding extrema of functions are given in the form of ODEs whose equilibria are identical with the extrema being sought. In fact, linear and quadratic programming problems are solved using the class of ODEs called gradient dynamical systems (GDSs). These optimization problems are transformed into unconstrained optimization problems using exact penalty functions, which are then solved using a gradient descent method, i.e., a GDS. The key idea is to adjust the penalty parameters, interpreted as control parameters, in the resulting GDS, using a control Liapunov function (CLF) approach. Since the system being studied is a GDS, the energy or potential function whose gradient defines the right-hand side (of the GDS) is a natural Liapunov function. The fact that these GDFSs can be represented and implemented as a class of recurrent neural networks explains the neologism neural-gradient dynamical systems that occurs in the chapter title, essentially to call the reader's attention to the equivalence between a GDS approach to optimization and the recurrent neural network approach. Anovelty that arises in the approach presented in this chapter is that nonsmooth CLFs have to be used since exact penalty functions are nonsmooth (i.e., have finitely many points of nondifferentiability). This technicality is handled using a generalized Persidskii-type result, presented in Chapter 1, that allows the treatment of a class of differential equations with discontinuous right-hand sides. Previous chapter Next chapter RelatedDetails Published:2006ISBN:978-0-89871-602-3eISBN:978-0-89871-866-9 https://doi.org/10.1137/1.9780898718669Book Series Name:Advances in Design and ControlBook Code:DC10Book Pages:xxv + 265Key words:Control theory, system theory, iterative methods, numerical methods for ODEs, Liapunov functions, gradient dynamical systems

Neural Dynamics and Newton–Raphson Iteration for Nonlinear Optimization

A Novel Nonlinear Dynamic System Modeling Approach Using Radial Basis Function Neural Networks

DNA-NSGA-Ⅱ Nonlinear Dynamic System Modeling Approach Using RBF Neural Networks

An accelerated neural dynamics model for solving dynamic nonlinear optimization problem and its applications

A discrete-time neural network for optimization problems with hybrid constraints.

Robust Neural Differential Solution for Time-Dependent Nonlinear Optimization With Noises Rejection

Discrete-Time Zhang Neural Network for Online Time-Varying Nonlinear Optimization With Application to Manipulator Motion Generation

A fast neural hybrid Newton solver adapted to implicit methods for nonlinear dynamics

NGDE: A Niching-Based Gradient-Directed Evolution Algorithm for Nonconvex Optimization

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

Weathering of hair in trichoteiromania

Neural-Gradient Dynamical Systems for Linear and Quadratic Programming Problems

Enhancing neurodynamic approach with physics-informed neural networks for solving non-smooth convex optimization problems

An Adaptive Gradient Neural Network to Solve Dynamic Linear Matrix Equations

Second-order Neural Network Training Using Complex-step Directional Derivative

From Zhang Neural Network to Newton Iteration for Matrix Inversion

Efficient, Accurate and Stable Gradients for Neural ODEs

Generalized Gradient Projection Neural Networks for Nonsmooth Optimization Problems

TO‐NODE: Topology optimization with neural ordinary differential equation

Comparative studies and performance analysis on neural-dynamics-driven control of redundant robot manipulators with unknown models

Neural downscaling for complex systems: from large-scale to small-scale by neural operator