Abstract:In this paper, the infinite horizon optimal control problems for singular diffusion processes are considered from the viewpoints of Markov decision processes and perturbation analysis, where the singularity of diffusion means that the covariance matrix of the system noise is allowed to be degenerate. A formula of performance difference under two different controls is derived and leads to a comparison theorem. By the comparison theorem, starting from a control, a so-called better control can be selected. Therefore, a control policy iteration algorithm is developed, by which the performance improves step by step and converges to the optimal one. When this applies to the stochastic affine nonlinear regulator and stochastic linear quadratic optimal control problems, better control can be constructed in a closed form. It is also shown that when the considered stochastic systems degenerate to the deterministic ones, the proposed algorithm reduces to the adaptive dynamic programming algorithm [J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, Adaptive dynamic programming, IEEE Trans. Systems Man Cybernet., 32 (2002), pp. 140-153] for the affine nonlinear systems and to the well-known Kleinman algorithm [D. L. Kleinman, On an iterative technique for Riccati equation computation, IEEE Trans. Automat. Control, 13 (1968), pp. 114-115] for the linear quadratic optimal control problem.

On policy iteration‐based discounted optimal control

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Policy Iteration Algorithm for Constrained Cost Optimal Control of Discrete-Time Nonlinear System

Modified λ-Policy Iteration Based Adaptive Dynamic Programming for Unknown Discrete-Time Linear Systems

Adaptive optimal control for a class of nonlinear partially uncertain dynamic systems via policy iteration

Policy iteration for discrete-time systems with discounted costs: stability and near-optimality guarantees

Temporal Difference-Based Policy Iteration for Optimal Control of Stochastic Systems

Policy Iteration Based Feedback Control

Modified $\lambda$-Policy Iteration Based Adaptive Dynamic Programming for Unknown Discrete-Time Linear Systems

Bias-Policy Iteration-Based Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems

Policy Iteration Algorithm for Singular Controlled Diffusion Processes

Modified general policy iteration based adaptive dynamic programming for unknown discrete‐time linear systems

Multistep Look-Ahead Policy Iteration for Optimal Control of Discrete-Time Nonlinear Systems with Isoperimetric Constraints

Online Policy Iteration Algorithm for Optimal Control of Linear Hyperbolic PDE Systems

Model-Free $δ$-Policy Iteration Based on Damped Newton Method for Nonlinear Continuous-Time H$\infty$ Tracking Control

Continuous-Time Distributed Policy Iteration for Multicontroller Nonlinear Systems

AN OPEN-CLOSED-LOOP PI-TYPE ITERATIVE LEARNING CONTROL SCHEME FOR DISCRETE NONLINEAR TIME-VARYING SYSTEMS AND ITS CONVERGENCE

Policy Iteration-Based Learning Design for Linear Continuous-Time Systems Under Initial Stabilizing OPFB Policy

Error Bound Analysis of Q-Function for Discounted Optimal Control Problems With Policy Iteration.

A New Continuous-Time Policy Iteration for Time-Varying Nonlinear Systems

On Convergence Analysis of Policy Iteration Algorithms for Entropy-Regularized Stochastic Control Problems