Adaptive autonomous soaring of multiple UAVs using Simultaneous Perturbation Stochastic Approximation

Cathrine Antal,O. Granichin,Sergey Levi

DOI: https://doi.org/10.1109/CDC.2010.5717903

2010-12-01

Abstract:This paper presents a new algorithm for maximizing the flight duration of a single UAV (Uninhabited Air Vehicle) and UAVs group using the thermal model developed by Allen at NASA Dryden.

What problem does this paper attempt to address?

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Lingzhi Zhang,Lei Xie,Yi Jiang,Zhishan Li,Xueqin Liu,Hongye Su

DOI: https://doi.org/10.1109/tnnls.2023.3326397

IF: 11.8

2024-01-01

IEEE Transactions on Cybernetics

Abstract:The state and input constraints of nonlinear systems could greatly impede the realization of their optimal control when using reinforcement learning (RL)-based approaches since the commonly used quadratic utility functions cannot meet the requirements of solving constrained optimization problems. This article develops a novel optimal control approach for constrained discrete-time (DT) nonlinear systems based on safe RL. Specifically, a barrier function (BF) is introduced and incorporated with the value function to help transform a constrained optimization problem into an unconstrained one. Meanwhile, the minimum of such an optimization problem can be guaranteed to occur at the origin. Then a constrained policy iteration (PI) algorithm is developed to realize the optimal control of the nonlinear system and to enable the state and input constraints to be satisfied. The constrained optimal control policy and its corresponding value function are derived through the implementation of two neural networks (NNs). Performance analysis shows that the proposed control approach still retains the convergence and optimality properties of the traditional PI algorithm. Simulation results of three examples reveal its effectiveness.
Immersion and Invariance Adaptive Control of Nonlinearly Parameterized Nonlinear Systems

Xiangbin Liu,Romeo Ortega,Hongye Su,Jian Chu

DOI: https://doi.org/10.1109/tac.2010.2052389

2010-01-01

IFAC Proceedings Volumes

Abstract:A new framework to design adaptive controllers for nonlinearly parameterized systems is proposed in this paper. The key step is the construction of a monotone function, which explicitly depends on some of the estimator tuning parameters. Monotonicity—or the related property of convexity—have already been explored by several authors with convexity being an a priori assumption that is valid only on some region of state space. In our approach monotonicity is enforced by the designer, effectively becoming a synthesis tool. One consequence of this fact is that the controller does not rely on state—dependent switching. In order to dispose of degrees of freedom to render the function monotone we depart from standard adaptive control and adopt instead the recently introduced Immersion and Invariance approach.
Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

Qinmin Yang,S. Jagannathan

DOI: https://doi.org/10.1109/acc.2007.4282709

2007-01-01

Abstract:In this paper, a novel reinforcement learning neural network (NN)-based controller, referred to adaptive critic controller, is proposed for general multi-input and multi- output affine unknown nonlinear discrete-time systems in the presence of bounded disturbances. Adaptive critic designs consist of two entities, an action network that produces optimal solution and a critic that evaluates the performance of the action network. The critic is termed adaptive as it adapts itself to output the optimal cost-to-go function and the action network is adapted simultaneously based on the information from the critic. In our online learning method, one NN is designated as the critic NN, which approximates the Bellman equation. An action NN is employed to derive the control signal to track a desired system trajectory while minimizing the cost function. Online updating weight tuning schemes for these two NNs are also derived and uniformly ultimate boundedness (UUB) of the tracking error and weight estimates is shown. The effectiveness of the controller is evaluated on a two-link robotic arm system.
A Combined Policy Gradient and Q-learning Method for Data-driven Optimal Control Problems

Mingduo Lin,Derong Liu,Bo Zhao,Qionghai Dai,Yi Dong

DOI: https://doi.org/10.1109/icist.2019.8836932

2019-01-01

Abstract:This paper focuses on the data-driven controller design for optimal control problems of nonlinear nonaffine discrete-time systems. A novel policy gradient and Q-learning (PGQL) adaptive algorithm which learns the optimal control policy from real empirical data is developed without requiring system dynamics. A policy iteration scheme is designed to iteratively update the approximate Q-function, and the control policy is improved via gradient method until they converge to the bounded regions of the optimal Q-function and the optimal control policy, respectively. Two neural networks (NNs) are employed to realize the developed algorithm. Moreover, the convergence analysis of approximate Q-function is established. Since the control policy is parameterized, it can be upgraded through updating the actor-NN parameters in the direction of the performance gradient. Finally, the simulation results are given to verify the performance of the developed PGQL adaptive algorithm.
Iterative design of suboptimal feedback control for bilinear parabolic PDE systems

Chao Xu,Eugenio Schuster

DOI: https://doi.org/10.1109/ACC.2009.5160545

2009-01-01

Abstract:Optimal control of infinite dimensional systems is one of the central problems in the control of distributed parameter systems. With the development of high performance computers, numerical methods for optimal control design have regained attention and achieved significant progress, mostly in the form of open-loop solutions. We consider in this work an optimal control problem for a bilinear parabolic partial differential equation (PDE) system. Based on the optimality conditions derived from Pontryagin's maximum principle for a reduced-order model, and stated as a two-boundary-value problem, we propose an iterative scheme for suboptimal closed-loop control design. In each iteration step, we take advantage of linear synthesis methods to construct a sequence of controllers. The convergence of the controller sequence is proved in appropriate functional spaces. When compared with previous iterative schemes, the proposed scheme avoids repeated numerical computation of the Riccati equation and therefore reduces significantly the number of ODEs that must be solved at each iteration step. A numerical simulation study shows the effectiveness of this new approach.
Near Optimal Neural Network-based Output Feedback Control of Affine Nonlinear Discrete-Time Systems

Qinmin Yang,Sarangapani Jagannathan

DOI: https://doi.org/10.1109/isic.2007.4450950

2007-01-01

Abstract:In this paper, a novel online reinforcement learning neural network (NN)-based optimal output feedback controller, referred to as adaptive critic controller, is proposed for affine nonlinear discrete-time systems, to deliver a desired tracking performance. The adaptive critic design consist of three entities, an observer to estimate the system states, an action network that produces optimal control input and a critic that evaluates the performance of the action network. The critic is termed adaptive as it adapts itself to output the optimal cost-to-go function which is based on the standard Bellman equation. By using the Lyapunov approach, the uniformly ultimate boundedness (UUB) of the estimation and tracking errors and weight estimates is demonstrated. The effectiveness of the controller is evaluated for the task of nanomanipulation in a simulation environment.
Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Zhongpu Xia,Dongbin Zhao,Huajin Tang

DOI: https://doi.org/10.3182/20140824-6-za-1003.01368

2014-01-01

IFAC Proceedings Volumes

Abstract:Abstract In this paper, a model-free and effective approach is proposed to solve infinite horizon optimal control problem for affine nonlinear systems based on adaptive dynamic programming technique. The developed approach, referred to as the actor-critic structure, employs two multilayer perceptron neural networks to approximate the state-action value function and the control policy, respectively. It uses data collected arbitrarily from any reasonable sampling distribution for policy iteration. In the policy evaluation phase, a novel objective function is defined for updating the critic network, and thus makes the critic network converge to the Bellman equation directly rather than iteratively. In the policy improvement phase, the action network is updated to minimize the outputs of the critic network. The two phases alternate until no more improvement of the control policy is observed, such that the optimal control policy is achieved. Two simulation examples are provided to show the effectiveness of the approach.
Reinforcement Learning Controller Design for Affine Nonlinear Discrete-Time Systems Using Online Approximators

Qinmin Yang,Sarangapani Jagannathan

DOI: https://doi.org/10.1109/tsmcb.2011.2166384

2011-01-01

IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics)

Abstract:In this paper, reinforcement learning state- and output-feedback-based adaptive critic controller designs are proposed by using the online approximators (OLAs) for a general multi-input and multioutput affine unknown nonlinear discretetime systems in the presence of bounded disturbances. The proposed controller design has two entities, an action network that is designed to produce optimal signal and a critic network that evaluates the performance of the action network. The critic estimates the cost-to-go function which is tuned online using recursive equations derived from heuristic dynamic programming. Here, neural networks (NNs) are used both for the action and critic whereas any OLAs, such as radial basis functions, splines, fuzzy logic, etc., can be utilized. For the output-feedback counterpart, an additional NN is designated as the observer to estimate the unavailable system states, and thus, separation principle is not required. The NN weight tuning laws for the controller schemes are also derived while ensuring uniform ultimate boundedness of the closed-loop system using Lyapunov theory. Finally, the effectiveness of the two controllers is tested in simulation on a pendulum balancing system and a two-link robotic arm system.
Learning-based adaptive optimal control of linear time-delay systems: A value iteration approach

Leilei Cui,Bo Pang,Miroslav Krstić,Zhong-Ping Jiang

DOI: https://doi.org/10.1016/j.automatica.2024.111944

IF: 6.4

2024-10-05

Automatica

Abstract:This paper proposes a novel learning-based adaptive optimal controller design method for a class of continuous-time linear time-delay systems. A key strategy is to exploit the state-of-the-art reinforcement learning (RL) techniques and adaptive dynamic programming (ADP), and propose a data-driven method to learn the near-optimal controller without the precise knowledge of system dynamics. Specifically, a value iteration (VI) algorithm is proposed to solve the infinite-dimensional Riccati equation for the linear quadratic optimal control problem of time-delay systems using finite samples of input-state trajectory data. It is rigorously proved that the proposed VI algorithm converges to the near-optimal solution. Compared with the previous literature, the nice features of the proposed VI algorithm are that it is directly developed for continuous-time systems without discretization and an initial admissible controller is not required for implementing the algorithm. The efficacy of the proposed methodology is demonstrated by two practical examples of metal cutting and autonomous driving.

automation & control systems,engineering, electrical & electronic
Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

Miao Huang,Cong Liu,Xiaoqi He,Longhua Ma,Zheming Lu,Hongye Su

DOI: https://doi.org/10.1016/j.neucom.2020.03.061

IF: 6

2020-01-01

Neurocomputing

Abstract:In this work, output-feedback control problems for a class of discrete-time non-affine nonlinear systems with unknown control directions and input constraints are considered by using reinforcement learning (RL) method. Two neural networks (NNs) implement the control: 1) a critic NN that estimates a non-quadratic strategic utility function (SUF) and 2) an action NN that generates optimized control input and minimizes the SUF. The implicit function theorem is applied to obtain the optimal control law since the control is appeared in a non-affine form. For the first time, the discrete Nussbaum gain is introduced to overcome the difficulty that the control directions are unknown and a non-quadratic SUF is used to deal with the control constraints in the RL-based control. The theoretical derivation of the uniformly ultimately boundedness of the NN weights and the closed-loop output tracking error is given. And two numerical examples have been supplied to valid the proposed method.
A new Q‐function structure for model‐free adaptive optimal tracking control with asymmetric constrained inputs

Mingming Zhao,Ding Wang,Menghua Li,Ning Gao,Junfei Qiao

DOI: https://doi.org/10.1002/acs.3761

IF: 3.369

2024-02-06

International Journal of Adaptive Control and Signal Processing

Abstract:Summary This article aims to design a model‐free adaptive tracking controller for discrete‐time nonlinear systems with unknown dynamics and asymmetric control constraints. First, a new Q‐function structure is designed by introducing the control input into the tracking error of the next moment, in order to eliminate the final tracking error, avoid the steady control, and ignore the discount factor. Second, via system transformation, a general performance index is developed to overcome the challenge caused by asymmetric constraints of implicit control inputs. By this operation, the constrained tracking problem is converted to an unconstrained optimal tracking problem without the traditional nonquadratic performance function that is only applicable to explicit control inputs. Then, a value‐iteration‐based Q‐learning (VIQL) algorithm is derived to seek the optimal Q‐function and the optimal control policy by using offline data rather than the mathematical model. Next, the convergence, monotonicity, and stability properties of VIQL are investigated to demonstrate that the iterative Q‐function sequence can converge to the optimal Q‐function under ideal conditions. To realize the VIQL algorithm, the critic neural network is employed to approximate the Q‐function. Finally, simulation results and comparative experiments are conducted to demonstrate the validity and effectiveness of the present VIQL scheme.

automation & control systems,engineering, electrical & electronic
Control of Nonaffine Nonlinear Discrete-Time Systems Using Reinforcement-Learning-Based Linearly Parameterized Neural Networks

Qinmin Yang,Jonathan Blake Vance,S. Jagannathan

DOI: https://doi.org/10.1109/tsmcb.2008.926607

2008-01-01

Abstract:A nonaffine discrete-time system represented by the nonlinear autoregressive moving average with eXogenous input (NARMAX) representation with unknown nonlinear system dynamics is considered. An equivalent affinelike representation in terms of the tracking error dynamics is first obtained from the original nonaffine nonlinear discrete-time system so that reinforcement-learning-based near-optimal neural network (NN) controller can be developed. The control scheme consists of two linearly parameterized NNs. One NN is designated as the critic NN, which approximates a predefined long-term cost function, and an action NN is employed to derive a near-optimal control signal for the system to track a desired trajectory while minimizing the cost function simultaneously. The NN weights are tuned online. By using the standard Lyapunov approach, the stability of the closed-loop system is shown. The net result is a supervised actor-critic NN controller scheme which can be applied to a general nonaffine nonlinear discrete-time system without needing the affinelike representation. Simulation results demonstrate satisfactory performance of the controller.
Reinforcement learning based adaptive optimal control for constrained nonlinear system via a novel state-dependent transformation

Lei Yan,Zhi Liu,Yun Zhang,Zongze Wu,C.L. Philip Chen

DOI: https://doi.org/10.1016/j.isatra.2022.07.006

IF: 7.3

2022-07-15

ISA Transactions

Abstract:Highlights • A novel NSDF can achieve the following three points in de-signing adaptive control scheme is proposed to deal with state constraints: (a) feasibility condition is no longer to be required. (b) the cases that with or without state constraints can be covered simultaneously. (c) the steady-state tracking error is not affected by the proposed novel NSDF. • A reinforcement learning based adaptive optimal controller for strict feedback nonlinear systems subject to asymmetric time-varying state constraints is developed. Existing schemes for state-constrained systems either impose feasibility conditions or ignore the optimality. In this article, an adaptive optimal control scheme for the strict-feedback nonlinear system is proposed, which benefits from two design steps. Firstly, a novel nonlinear state-dependent function (NSDF) is formulated to equivalently transform the system into a non-constrained one to deal with state constraints without the requirements on feasibility conditions. Secondly, an adaptive optimal control scheme is designed for the non-constrained system, in which reinforcement learning (RL) is utilized to yield the optimal controller in each designing procedure. Updating rules of the actor and critic neural network are driven by the modified adaptive laws, used to approximate the optimal virtual and actual controllers. It is proved that all the signals in the closed-loop system are bounded and the output tracking error converges to an adjustable neighborhood of the origin not affected by the proposed NSDF. Two simulation examples are presented illustrating the effectiveness of the proposed scheme.

automation & control systems,instruments & instrumentation,engineering, multidisciplinary
Model-Free Optimal Tracking Design With Evolving Control Strategies via Q-Learning

Ding Wang,Haiming Huang,Mingming Zhao

DOI: https://doi.org/10.1109/tcsii.2024.3359258

2024-01-01

Abstract:This paper leverages a value-iteration-based Q-learning (VIQL) scheme to tackle optimal tracking problems for nonlinear nonaffine systems. The optimal policy is learned from measured data instead of a precise mathematical model. Furthermore, a novel criterion is proposed to determine the stability of the iterative policy based on measured data. The evolving control algorithm is developed to verify the proposed criterion by employing these stable policies for system control. The advantage of the early elimination of tracking errors is provided by this approach since various stable policies can be employed before obtaining the optimal strategy. Finally, the effectiveness of the developed algorithm is demonstrated by a simulation experiment.

engineering, electrical & electronic
Inverse Optimal Adaptive Neural Control for State-Constrained Nonlinear Systems

Kaixin Lu,Zhi Liu,Haoyong Yu,C. L. Philip Chen,Yun Zhang

DOI: https://doi.org/10.1109/tnnls.2023.3243084

IF: 14.255

2023-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Optimizing a performance objective during control operation while also ensuring constraint satisfactions at all times is important in practical applications. Existing works on solving this problem usually require a complicated and time-consuming learning procedure by employing neural networks, and the results are only applicable for simple or time-invariant constraints. In this work, these restrictions are removed by a newly proposed adaptive neural inverse approach. In our approach, a new universal barrier function, which is able to handle various dynamic constraints in a unified manner, is proposed to transform the constrained system into an equivalent one with no constraint. Based on this transformation, a switched-type auxiliary controller and a modified criterion for inverse optimal stabilization are proposed to design an adaptive neural inverse optimal controller. It is proven that optimal performance is achieved with a computationally attractive learning mechanism, and all the constraints are never violated. Besides, improved transient performance is obtained in the sense that the bound of the tracking error could be explicitly designed by users. An illustrative example verifies the proposed methods.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Reinforcement Learning-Based Control for a Class of Nonlinear Systems with Unknown Control Directions

Xiaoling Song,Miao Huang,Gang Wen,Longhua Ma,Jiaqing Yao,Zheming Lu

DOI: https://doi.org/10.23919/chicc.2019.8865292

2019-01-01

Abstract:In this work, output-feedback control problems for a class of discrete-time non-affine nonlinear systems with unknown control directions are considered by using reinforcement learning (RL) method. Two neural networks (NNs) implement the control: 1) a critic NN that estimates a cost function and 2) an action NN that generates optimized control input. The implicit function theorem is applied to obtain the optimal control law since the control is appeared in a non-affine form. The discrete Nussbaum gain is introduced to overcome the difficulty that the control directions are unknown in the RL-based adaptive control. Numerical examples have been supplied to valid the proposed method.
Adaptive Optimal Control with Guaranteed Convergence Rate for Continuous-Time Linear Systems with Completely Unknown Dynamics.

Kai Zhang,Suo-Liang Ge

DOI: https://doi.org/10.1109/access.2019.2892427

IF: 3.9

2019-01-01

IEEE Access

Abstract:The traditional linear quadratic optimal control can be summarized as finding the state feedback controller so that the closed-loop system is stable and the performance index is minimum. And, it is well-known that the solution of the linear quadratic optimal control problem can be solved by algebraic Riccati equation. However, the feature of the traditional linear quadratic optimal control theory is that the convergence rate is not specified. This may result in the phenomena of slow convergence. In this paper, we mainly consider the linear quadratic optimal control with guaranteed convergence rate (LQOCGCR) and propose a policy iteration-based adaptive dynamic programming algorithm that includes offline and online versions for finding the solution of the LQOCGCR. Finally, a numerical example is worked out to show the effectiveness of the proposed approach.
The intelligent critic framework for advanced optimal control

Ding Wang,Mingming Ha,Mingming Zhao

DOI: https://doi.org/10.1007/s10462-021-10118-9

IF: 9.588

2022-01-01

Artificial Intelligence Review

Abstract:The idea of optimization can be regarded as an important basis of many disciplines and hence is extremely useful for a large number of research fields, particularly for artificial-intelligence-based advanced control design. Due to the difficulty of solving optimal control problems for general nonlinear systems, it is necessary to establish a kind of novel learning strategies with intelligent components. Besides, the rapid development of computer and networked techniques promotes the research on optimal control within discrete-time domain. In this paper, the bases, the derivation, and recent progresses of critic intelligence for discrete-time advanced optimal control design are presented with an emphasis on the iterative framework. Among them, the so-called critic intelligence methodology is highlighted, which integrates learning approximators and the reinforcement formulation.

computer science, artificial intelligence
NN-Based Reinforcement Learning Optimal Control for Inequality-Constrained Nonlinear Discrete-Time Systems With Disturbances

Shu Li,Liang Ding,Miao Zheng,Zixuan Liu,Xinyu Li,Huaiguang Yang,Haibo Gao,Zongquan Deng

DOI: https://doi.org/10.1109/tnnls.2023.3287881

IF: 14.255

2023-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Based on actor-critic neural networks (NNs), an optimal controller is proposed for solving the constrained control problem of an affine nonlinear discrete-time system with disturbances. The actor NNs provide the control signals and the critic NNs work as the performance indicators of the controller. By converting the original state constraints into new input constraints and state constraints, the penalty functions are introduced into the cost function, and then the constrained optimal control problem is transformed into an unconstrained one. Further, the relationship between the optimal control input and worst-case disturbance is obtained using the Game theory. With Lyapunov stability theory, the control signals are ensured to be uniformly ultimately bounded (UUB). Finally, the effectiveness of the control algorithms is tested through a numeral simulation using a third-order dynamic system.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Online Adaptive Optimal Control for Continuous-Time Nonlinear Systems with Completely Unknown Dynamics.

Yongfeng Lv,Jing Na,Qinmin Yang,Xing Wu,Yu Guo

DOI: https://doi.org/10.1080/00207179.2015.1060362

IF: 2.102

2015-01-01

International Journal of Control

Abstract:An online adaptive optimal control is proposed for continuous-time nonlinear systems with completely unknown dynamics, which is achieved by developing a novel identifier-critic-based approximate dynamic programming algorithm with a dual neural network (NN) approximation structure. First, an adaptive NN identifier is designed to obviate the requirement of complete knowledge of system dynamics, and a critic NN is employed to approximate the optimal value function. Then, the optimal control law is computed based on the information from the identifier NN and the critic NN, so that the actor NN is not needed. In particular, a novel adaptive law design method with the parameter estimation error is proposed to online update the weights of both identifier NN and critic NN simultaneously, which converge to small neighbourhoods around their ideal values. The closed-loop system stability and the convergence to small vicinity around the optimal solution are all proved by means of the Lyapunov theory. The proposed adaptation algorithm is also improved to achieve finite-time convergence of the NN weights. Finally, simulation results are provided to exemplify the efficacy of the proposed methods.

Adaptive autonomous soaring of multiple UAVs using Simultaneous Perturbation Stochastic Approximation

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Immersion and Invariance Adaptive Control of Nonlinearly Parameterized Nonlinear Systems

Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

A Combined Policy Gradient and Q-learning Method for Data-driven Optimal Control Problems

Iterative design of suboptimal feedback control for bilinear parabolic PDE systems

Near Optimal Neural Network-based Output Feedback Control of Affine Nonlinear Discrete-Time Systems

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Reinforcement Learning Controller Design for Affine Nonlinear Discrete-Time Systems Using Online Approximators

Learning-based adaptive optimal control of linear time-delay systems: A value iteration approach

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

A new Q‐function structure for model‐free adaptive optimal tracking control with asymmetric constrained inputs

Control of Nonaffine Nonlinear Discrete-Time Systems Using Reinforcement-Learning-Based Linearly Parameterized Neural Networks

Reinforcement learning based adaptive optimal control for constrained nonlinear system via a novel state-dependent transformation

Model-Free Optimal Tracking Design With Evolving Control Strategies via Q-Learning

Inverse Optimal Adaptive Neural Control for State-Constrained Nonlinear Systems

Reinforcement Learning-Based Control for a Class of Nonlinear Systems with Unknown Control Directions

Adaptive Optimal Control with Guaranteed Convergence Rate for Continuous-Time Linear Systems with Completely Unknown Dynamics.

The intelligent critic framework for advanced optimal control

NN-Based Reinforcement Learning Optimal Control for Inequality-Constrained Nonlinear Discrete-Time Systems With Disturbances

Online Adaptive Optimal Control for Continuous-Time Nonlinear Systems with Completely Unknown Dynamics.