Policy iteration based feedback control

ZhangKan-Jian,XuYan-Kai,ChenXi,CaoXi-Ren

IF: 6.4

2008-01-01

Automatica

Abstract:It is well known that stochastic control systems can be viewed as Markov decision processes (MDPs) with continuous state spaces. In this paper, we propose to apply the policy iteration approach in ...

What problem does this paper attempt to address?

Policy Iteration Based Feedback Control

Kan-Jian Zhang,Yan-Kai Xu,Chen B. Xi,Xi-Ren Cao

DOI: https://doi.org/10.1016/j.automatica.2007.08.014

IF: 6.4

2007-01-01

Automatica

Abstract:It is well known that stochastic control systems can be viewed as Markov decision processes (MDPs) with continuous state spaces. In this paper, we propose to apply the policy iteration approach in MDPs to the optimal control problem of stochastic systems. We first provide an optimality equation based on performance potentials and develop a policy iteration procedure. Then we apply policy iteration to the jump linear quadratic problem and obtain the coupled Riccati equations for their optimal solutions. The approach is applicable to linear as well as nonlinear systems and can be implemented on-line on real world systems without identifying all the system structure and parameters.
Feedback Control of Nonlinear Stochastic Dynamic Systems for Accurately Tracking a Specified Stationary Probability Density Function

Lei Xia,Zhiqiang Liu,Ronghua Huan,Weiqiu Zhu

DOI: https://doi.org/10.1177/10775463211068904

IF: 2.633

2022-01-01

Journal of Vibration and Control

Abstract:Limited by the difficulty of obtaining the exact stationary solution of Fokker–Planck–Kolmogorov (FPK) equation, the general control strategy for tracking a specified stationary probability density function have not been obtained till now. However, the feedback control for tracking a pre-specified stationary probability density function plays an important role in engineering or industrial systems. In this paper, a feedback control strategy of nonlinear stochastic dynamic system for accurately tracking a specified stationary probability density function without the requirement of exact stationary solution of Fokker–Planck–Kolmogorov equation is proposed. According to the probability conservation form of the Fokker–Planck–Kolmogorov equation, the stationary Fokker–Planck–Kolmogorov equation is split into a probability circulation flow (PCF) equation and a probability potential flow (PPF) equation. The control force is divided into probability circulation flow part and probability potential flow part accordingly. The probability circulation flow part of the control force is determined to satisfy the probability circulation flow term of the controlled system constructed from the target stationary probability density function. The probability potential flow part of the control force is obtained by solving the probability potential flow equation. A two-dimensional nonlinear stochastic system is carried out as an example. The control force is designed to track different types of target stationary probability density functions. Numerical results show that the proposed control strategy can accurately track the stationary probability density functions without the requirement of the exact solution of Fokker–Planck–Kolmogorov equation. The control efficiency can be regulated by the control parameter C.
A policy iteration algorithm for non-Markovian control problems

Dylan Possamaï,Ludovic Tangpi

2024-09-06

Abstract:In this paper, we propose a new policy iteration algorithm to compute the value function and the optimal controls of continuous time stochastic control problems. The algorithm relies on successive approximations using linear-quadratic control problems which can all be solved explicitly, and only require to solve recursively linear PDEs in the Markovian case. Though our procedure fails in general to produce a non-decreasing sequence like the standard algorithm, it can be made arbitrarily close to being monotone. More importantly, we recover the standard exponential speed of convergence for both the value and the controls, through purely probabilistic arguments which are significantly simpler than in the classical case. Our proof also accommodates non-Markovian dynamics as well as volatility control, allowing us to obtain the first convergence results in the latter case for a state process in multi-dimensions.

Optimization and Control,Probability
Feedback nonlinear discrete-time systems

Miao Yu,Jiasen Wang,Donglian Qi

DOI: https://doi.org/10.1080/00207721.2013.767950

IF: 2.648

2014-01-01

International Journal of Systems Science

Abstract:In this paper, we design an adaptive iterative learning control method for a class of high-order nonlinear output feedback discrete-time systems with random initial conditions and iteration-varying desired trajectories. An n-step ahead predictor approach is employed to estimate future outputs. The discrete Nussbaum gain method is incorporated into the control design to deal with unknown control directions. The proposed control algorithm ensures that the tracking error converges to zero asymptotically along the iterative learning axis except for the beginning outputs affected by random initial conditions. A numerical simulation is carried out to demonstrate the efficacy of the presented control laws.
Approximate Policy Iteration for Robust Stochastic Control of Multi-agent Markov Decision Processes

Feng Huang,Ming Cao,Long Wang

DOI: https://doi.org/10.1109/tac.2024.3510596

IF: 6.549

2024-01-01

IEEE Transactions on Automatic Control

Abstract:In stochastic dynamic environments, multi-agent Markov decision processes have emerged as a versatile paradigm for studying sequential decision-making problems of fully cooperative multi-agent systems. However, the optimality of the derived policies is usually sensitive to model parameters, which are typically unknown and required to be estimated from noisy data in practice. To investigate the sensitivity of optimal policies to these uncertain parameters, we study a robust stochastic control problem of multi-agent Markov decision processes where all agents constitute a centralized controller whose goal is to seek a maximal long-term return of all agents and the uncertainty plays a role of disturbance for achieving this goal, and provide a solution concept of robust team optimality for decisions of all agents. To seek such a solution, we develop a robust iterative learning algorithm of policies for all agents and present its convergence analysis. This algorithm, compared with robust dynamic programming, not only possesses a faster convergence rate, but also allows for using approximation calculations to alleviate required computational resources. Moreover, some numerical simulations are presented to demonstrate the effectiveness of the algorithm by extending the model of sequential social dilemmas to uncertain scenarios.
On policy iteration‐based discounted optimal control

Botao Dong,Longyang Huang,Xiwen Ma,Weidong Zhang

DOI: https://doi.org/10.1002/rnc.7245

IF: 3.8973

2024-02-17

International Journal of Robust and Nonlinear Control

Abstract:This article investigates the policy iteration (PI) method for the discounted optimal control (DOC) problem of continuous‐time linear systems. We show the properties and convergence of the PI method. The theory analysis shows that the convergence of PI can be ensured without requiring the initial admissible control gain. The convergence rate of the PI method is provided. An iteration‐termination criterion is established for detecting the stability of the closed‐loop system under the control gain obtained by executing PI. Two kinds of data‐driven implementations are constructed without using prior information of the system dynamics. A simulation example is presented to validated the properties of the PI method.

automation & control systems,engineering, electrical & electronic,mathematics, applied
Policy iteration for parameterized Markov decision processes and its application

Li Xia,Qingshan Jia

DOI: https://doi.org/10.1109/ASCC.2013.6606023

2013-01-01

Abstract:In a parameterized Markov decision process (MDP), the decision maker has to choose the optimal parameters which induce the maximal average system reward. However, the traditional policy iteration algorithm is usually inapplicable because the parameters choosing is not independent of the system state. In this paper, we use the direct comparison approach to study this problem. A general difference equation is derived to compare the performance difference under different parameters. We derive a theoretical condition that can guarantee the application of policy iteration to the parameterized MDP. This policy iteration type algorithm is much more efficient than the gradient optimization algorithm for parameterized MDP. Finally, we study the service rate control problem of closed Jackson networks as an example to demonstrate the main idea of this paper.
Feedback Control Optimization for Stochastic Dynamic Systems under Wide-Band Random Excitations

LI Xue-ping,YING Zu-guang

DOI: https://doi.org/10.3969/j.issn.1004-4523.2007.06.016

2007-01-01

Abstract:A strategy of feedback control optimization of stochastic dynamic systems via stationary response is proposed.First,the control system subjected to wide-band random excitations is formulated based on the stochastic averaging method for quasi-integrable Hamiltonian systems and then transformed into Ito^ stochastic differential equation.Second,the averaged Ito^ stochastic differential equation for the system is derived and the stationary solution of the Fokker-Plank-Kolmogorov equation associated with the averaged Ito^ equation is obtained.Third,the mean square value of the control force and the Hamiltonian function are combined handily as the performance index of the system and the optimal control force is obtained.Finally,two examples are worked out in detail to illustrate the application and the effectiveness of the proposed strategy.
Continuous-Time Distributed Policy Iteration for Multicontroller Nonlinear Systems

Qinglai Wei,Hongyang Li,Xiong Yang,Haibo He

DOI: https://doi.org/10.1109/tcyb.2020.2979614

IF: 11.8

2021-05-01

IEEE Transactions on Cybernetics

Abstract:In this article, a novel distributed policy iteration algorithm is established for infinite horizon optimal control problems of continuous-time nonlinear systems. In each iteration of the developed distributed policy iteration algorithm, only one controller's control law is updated and the other controllers' control laws remain unchanged. The main contribution of the present algorithm is to improve the iterative control law one by one, instead of updating all the control laws in each iteration of the traditional policy iteration algorithms, which effectively releases the computational burden in each iteration. The properties of distributed policy iteration algorithm for continuous-time nonlinear systems are analyzed. The admissibility of the present methods has also been analyzed. Monotonicity, convergence, and optimality have been discussed, which show that the iterative value function is nonincreasingly convergent to the solution of the Hamilton–Jacobi–Bellman equation. Finally, numerical simulations are conducted to illustrate the effectiveness of the proposed method.

automation & control systems,computer science, cybernetics, artificial intelligence
Policy Iteration-Based Learning Design for Linear Continuous-Time Systems Under Initial Stabilizing OPFB Policy

Chengye Zhang,Ci Chen,Frank L Lewis,Shengli Xie

DOI: https://doi.org/10.1109/TCYB.2024.3418190

2024-07-22

Abstract:Policy iteration (PI), an iterative method in reinforcement learning, has the merit of interactions with a little-known environment to learn a decision law through policy evaluation and improvement. However, the existing PI-based results for output-feedback (OPFB) continuous-time systems relied heavily on an initial stabilizing full state-feedback (FSFB) policy. It thus raises the question of violating the OPFB principle. This article addresses such a question and establishes the PI under an initial stabilizing OPFB policy. We prove that an off-policy Bellman equation can transform any OPFB policy into an FSFB policy. Based on this transformation property, we revise the traditional PI by appending an additional iteration, which turns out to be efficient in approximating the optimal control under the initial OPFB policy. We show the effectiveness of the proposed learning methods through theoretical analysis and a case study.
Online policy iteration algorithm for semi-Markov switching state-space control processes

Qi Jiang,Hongsheng Xi,Bao-Qin Yin

DOI: https://doi.org/10.1109/CDC.2009.5400958

2009-01-01

Abstract:An event-based online policy iteration algorithm is presented for addressing hierarchical optimization problems. First, an event-driven analytical model with dynamic hierarchy called semi-Markov switching state-space control processes is introduced. Then, by exploiting the structure of dynamic hierarchy and the features of event-driven policy, an online adaptive optimization algorithm that combines potentials estimation and policy iteration is proposed. The convergence of this algorithm is also proved. Finally, as an illustrative example, the dynamic service composition in a service overlay network is formulated and addressed. Simulation results demonstrate the effectiveness of the presented algorithm.
Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Zhongpu Xia,Dongbin Zhao,Huajin Tang

DOI: https://doi.org/10.3182/20140824-6-za-1003.01368

2014-01-01

IFAC Proceedings Volumes

Abstract:Abstract In this paper, a model-free and effective approach is proposed to solve infinite horizon optimal control problem for affine nonlinear systems based on adaptive dynamic programming technique. The developed approach, referred to as the actor-critic structure, employs two multilayer perceptron neural networks to approximate the state-action value function and the control policy, respectively. It uses data collected arbitrarily from any reasonable sampling distribution for policy iteration. In the policy evaluation phase, a novel objective function is defined for updating the critic network, and thus makes the critic network converge to the Bellman equation directly rather than iteratively. In the policy improvement phase, the action network is updated to minimize the outputs of the critic network. The two phases alternate until no more improvement of the control policy is observed, such that the optimal control policy is achieved. Two simulation examples are provided to show the effectiveness of the approach.
Policy Iteration for Multiplicative Noise Output Feedback Control

Benjamin Gravell,Matilde Gargiani,John Lygeros,Tyler H. Summers

DOI: https://doi.org/10.48550/arXiv.2203.17165

2022-03-31

Systems and Control

Abstract:We propose a policy iteration algorithm for solving the multiplicative noise linear quadratic output feedback design problem. The algorithm solves a set of coupled Riccati equations for estimation and control arising from a partially observable Markov decision process (POMDP) under a class of linear dynamic control policies. We show in numerical experiments far faster convergence than a value iteration algorithm, formerly the only known algorithm for solving this class of problem. The results suggest promising future research directions for policy optimization algorithms in more general POMDPs, including the potential to develop novel approximate data-driven approaches when model parameters are not available.
Optimal Nonlinear Feedback Control of Quasi-Hamiltonian Systems

Weiqiu Zhu,Zuguang Ying

DOI: https://doi.org/10.1007/bf02875989

1999-01-01

Science in China Series A Mathematics

Abstract:An innovative strategy for optimal nonlinear feedback control of linear or nonlinear stochastic dynamic systems is proposed based on the stochastic averaging method for quasi-Hamiltonian systems and stochastic dynamic programming principle. Feedback control forces of a system are divided into conservative parts and dissipative parts. The conservative parts are so selected that the energy distribution in the controlled system is as requested as possible. Then the response of the system with known conservative control forces is reduced to a controlled diffusion process by using the stochastic averaging method. The dissipative parts of control forces are obtained from solving the stochastic dynamic programming equation.
From Optimization to Control: Quasi Policy Iteration

Mohammad Amin Sharifi Kolarijani,Peyman Mohajerin Esfahani

2024-10-14

Abstract:Recent control algorithms for Markov decision processes (MDPs) have been designed using an implicit analogy with well-established optimization algorithms. In this paper, we review this analogy across four problem classes with a unified solution characterization allowing for a systematic transformation of algorithms from one domain to the other. In particular, we identify equivalent optimization and control algorithms that have already been pointed out in the existing literature, but mostly in a scattered way. With this unifying framework in mind, we adopt the quasi-Newton method from convex optimization to introduce a novel control algorithm coined as quasi-policy iteration (QPI). In particular, QPI is based on a novel approximation of the "Hessian" matrix in the policy iteration algorithm by exploiting two linear structural constraints specific to MDPs and by allowing for the incorporation of prior information on the transition probability kernel. While the proposed algorithm has the same computational complexity as value iteration, it interestingly exhibits an empirical convergence behavior similar to policy iteration with a very low sensitivity to the discount factor.

Optimization and Control,Machine Learning,Systems and Control
Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

Bo Pang,Zhong-Ping Jiang

DOI: https://doi.org/10.48550/arXiv.2107.07788

2021-12-05

Abstract:This paper studies the adaptive optimal stationary control of continuous-time linear stochastic systems with both additive and multiplicative noises, using reinforcement learning techniques. Based on policy iteration, a novel off-policy reinforcement learning algorithm, named optimistic least-squares-based policy iteration, is proposed which is able to find iteratively near-optimal policies of the adaptive optimal stationary control problem directly from input/state data without explicitly identifying any system matrices, starting from an initial admissible control policy. The solutions given by the proposed optimistic least-squares-based policy iteration are proved to converge to a small neighborhood of the optimal solution with probability one, under mild conditions. The application of the proposed algorithm to a triple inverted pendulum example validates its feasibility and effectiveness.

Systems and Control,Machine Learning,Optimization and Control
Policy Iteration Reinforcement Learning Method for Continuous-Time Linear-Quadratic Mean-Field Control Problems

Na Li,Xun Li,Zuo Quan Xu

2024-11-04

Abstract:This paper employs a policy iteration reinforcement learning (RL) method to study continuous-time linear-quadratic mean-field control problems in infinite horizon. The drift and diffusion terms in the dynamics involve the states, the controls, and their conditional expectations. We investigate the stabilizability and convergence of the RL algorithm using a Lyapunov Recursion. Instead of solving a pair of coupled Riccati equations, the RL technique focuses on strengthening an auxiliary function and the cost functional as the objective functions and updating the new policy to compute the optimal control via state trajectories. A numerical example sheds light on the established theoretical results.

Optimization and Control
On the Feedback Law in Stochastic Optimal Nonlinear Control

Mohamed Naveed Gul Mohamed,Suman Chakravorty,Raman Goyal,Ran Wang

2024-03-26

Abstract:We consider the problem of nonlinear stochastic optimal control. This problem is thought to be fundamentally intractable owing to Bellman's ``curse of dimensionality". We present a result that shows that repeatedly solving an open-loop deterministic problem from the current state with progressively shorter horizons, similar to Model Predictive Control (MPC), results in a feedback policy that is $O(\epsilon^4)$ near to the true global stochastic optimal policy, \nxx{where $\epsilon$ is a perturbation parameter modulating the noise.} We show that the optimal deterministic feedback problem has a perturbation structure in that higher-order terms of the feedback law do not affect lower-order terms, and that this structure is lost in the optimal stochastic feedback problem. Consequently, solving the Stochastic Dynamic Programming problem is highly susceptible to noise, even when tractable, and in practice, the MPC-type feedback law offers superior performance even for stochastic systems.

Systems and Control,Robotics
State Feedback Control of Stochastic Hybrid Systems

Ying Yang

2010-01-01

Abstract:In this paper,the problem of stabilization is investigated for a class of Markovian switching stochastic hybrid systems.The switching parameters considered here form a finite homogeneous Markov chain.At first,by means of constructing a stochastic hybrid Lyapunov function,sufficient conditions are derived for the asymptotic stability of the overall system.Then,the feedback control is realized by the introduction of the adjustable transition probability and the control of both the Markov chain and the subsystems.
Reinforcement Learning Policies in Continuous-Time Linear Systems

Mohamad Kazem Shirani Faradonbeh,Mohamad Sadegh Shirani Faradonbeh

2023-06-08

Abstract:Linear dynamical systems that obey stochastic differential equations are canonical models. While optimal control of known systems has a rich literature, the problem is technically hard under model uncertainty and there are hardly any results. We initiate study of this problem and aim to learn (and simultaneously deploy) optimal actions for minimizing a quadratic cost function. Indeed, this work is the first that comprehensively addresses the crucial challenge of balancing exploration versus exploitation in continuous-time systems. We present online policies that learn optimal actions fast by carefully randomizing the parameter estimates, and establish their performance guarantees: a regret bound that grows with square-root of time multiplied by the number of parameters. Implementation of the policy for a flight-control task demonstrates its efficacy. Further, we prove sharp stability results for inexact system dynamics and tightly specify the infinitesimal regret caused by sub-optimal actions. To obtain the results, we conduct a novel eigenvalue-sensitivity analysis for matrix perturbation, establish upper-bounds for comparative ratios of stochastic integrals, and introduce the new method of policy differentiation. Our analysis sheds light on fundamental challenges in continuous-time reinforcement learning and suggests a useful cornerstone for similar problems.

Systems and Control,Machine Learning

Policy iteration based feedback control

Policy Iteration Based Feedback Control

Feedback Control of Nonlinear Stochastic Dynamic Systems for Accurately Tracking a Specified Stationary Probability Density Function

A policy iteration algorithm for non-Markovian control problems

Feedback nonlinear discrete-time systems

Approximate Policy Iteration for Robust Stochastic Control of Multi-agent Markov Decision Processes

On policy iteration‐based discounted optimal control

Policy iteration for parameterized Markov decision processes and its application

Feedback Control Optimization for Stochastic Dynamic Systems under Wide-Band Random Excitations

Continuous-Time Distributed Policy Iteration for Multicontroller Nonlinear Systems

Policy Iteration-Based Learning Design for Linear Continuous-Time Systems Under Initial Stabilizing OPFB Policy

Online policy iteration algorithm for semi-Markov switching state-space control processes

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Policy Iteration for Multiplicative Noise Output Feedback Control

Optimal Nonlinear Feedback Control of Quasi-Hamiltonian Systems

From Optimization to Control: Quasi Policy Iteration

Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

Policy Iteration Reinforcement Learning Method for Continuous-Time Linear-Quadratic Mean-Field Control Problems

On the Feedback Law in Stochastic Optimal Nonlinear Control

State Feedback Control of Stochastic Hybrid Systems

Reinforcement Learning Policies in Continuous-Time Linear Systems