An Introductory Review of Time-Inconsistent Stochastic Optimal Control

Yuan-Hua Ni,Ji-Feng Zhang,Georgi M. Dimirovski

DOI: https://doi.org/10.23919/chicc.2018.8483660

2018-01-01

Abstract:This paper gives an introductory review of time-inconsistent stochastic optimal control. We mainly explain the reasons that result in the time inconsistency, and review the recent progresses in time-inconsistent stochastic linear-quadratic optimal control.

What problem does this paper attempt to address?

Time-Inconsistent Discrete-Time Stochastic Linear-Quadratic Optimal Control : Time-consistent Solutions

Xun Li,Yuan-Hua Ni,Ji-Feng Zhang

2015-01-01

Abstract:In this paper, the time-consistent solutions of a timeinconsistent discrete-time stochastic linear-quadratic optimal control are investigated. Different from the existing literature, the definiteness constraint is not posed on the state and the control weight matrices of the cost functional. Necessary and sufficient conditions are, respectively, obtained to the existence of the open-loop time-consistent equilibrium control and the closedloop time-consistent equilibrium strategy, which contain the solvability of certain system of forward-backward stochastic difference equations, the stationary conditions and the convexity conditions. Furthermore, for the case that the system matrices are independent of the initial time, the existence of the openloop equilibrium pair for all the initial pairs is shown to be equivalent to the solvability of a system of generalized difference Riccati equations and a system of linear difference equations. Moreover, the existence of the closed-loop equilibrium strategy is shown to be equivalent to the solvability of another system of generalized difference Riccati equations. Interestingly, if solvable, the system of generalized difference Riccati equations for the open-loop formulation does not admit symmetric solution, while the one for the closed-loop formulation has symmetric solution.
Discrete-Time Stochastic Linear-Quadratic Optimal Control with Time-Inconsistency

Xun Li,Yuan-Hua Ni,Ji-Feng Zhang

DOI: https://doi.org/10.1016/j.ifacol.2015.12.210

2015-01-01

IFAC-PapersOnLine

Abstract:In this paper, the time-consistent solution of a discrete-time time-inconsistent stochastic linear-quadratic optimal control is investigated. Different from existing literature, the definiteness constraint is not posed on the state weight matrices and the control weight matrices of the cost functional. Necessary and suffcient conditions are obtained to the existence of the open-loop time-consistent equilibrium control, which contain the solvability of certain forward-backward stochastic difference equation systems, the stationary conditions and the convexity conditions. Under additional conditions, the closed-form of the open-loop equilibrium control is characterized via the solutions of systems of certain generalized difference Riccati equations. Interestingly, the system of generalized difference Riccati equations do not admit symmetry structure. Finally, for a special case of the considered problem, the existence of the open-loop equilibrium control for all the initial pairs is shown to be equivalent to the solvability of certain generalized difference Riccati equation.
On Time-Consistent Solution to Time-Inconsistent Linear-Quadratic Optimal Control of Discrete-Time Stochastic Systems

Xun Li,Yuan-Hua Ni,Ji-Feng Zhang

DOI: https://doi.org/10.48550/arXiv.1703.01942

2017-03-07

Abstract:In this paper, we investigate a class of time-inconsistent discrete-time stochastic linear-quadratic optimal control problems, whose time-consistent solutions consist of an open-loop equilibrium control and a linear feedback equilibrium strategy. The open-loop equilibrium control is defined for a given initial pair, while the linear feedback equilibrium strategy is defined for all the initial pairs. Maximum-principle-type necessary and sufficient conditions containing stationary and convexity are derived for the existence of these two time-consistent solutions, respectively. Furthermore, for the case where the system matrices are independent of the initial time, we show that the existence of the open-loop equilibrium control for a given initial pair is equivalent to the solvability of a set of nonsymmetric generalized difference Riccati equations and a set of linear difference equations. Moreover, the existence of linear feedback equilibrium strategy is equivalent to the solvability of another set of symmetric generalized difference Riccati equations.

Optimization and Control
Stochastic Fractional Optimal Control of Quasi-Integrable Hamiltonian System with Fractional Derivative Damping

F. Hu,W. Q. Zhu,L. C. Chen

DOI: https://doi.org/10.1007/s11071-012-0547-3

IF: 6.549

2013-01-01

IEEE Transactions on Automatic Control

Abstract:In the present survey, some progress in the stochastic dynamics and fractional optimal control of quasi integrable Hamiltonian systems with fractional derivative damping is reviewed. First, the stochastic averaging method for quasi integrable Hamiltonian systems with fractional derivative damping under various random excitations is briefly introduced. Then, the stochastic stability, stochastic bifurcation, first passage time and reliability, and stochastic fractional optimal control of the systems studied by using the stochastic averaging method are summarized. The focus is placed on the effects of fractional derivative order on the dynamics and control of the systems. Finally, some possible extensions are pointed out.
Time-inconsistent Mean-Field Stochastic Linear-Quadratic Optimal Control

Ni Yuan-Hua

DOI: https://doi.org/10.1109/chicc.2016.7553752

2016-01-01

Abstract:This paper is concerned with a kind of time-consistent solution of the time-inconsistent mean-field stochastic linear-quadratic optimal control. Different from standard stochastic linear-quadratic problems, both the system matrices and the weighting matrices are dependent on initial time, and the conditional expectations of the control and state enter quadratically into the cost functional. Such features will ruin the Bellman optimality principle and result in the time-inconsistency of the optimal control. Due to the dynamical nature of control problems, a kind of open-loop time-consistent equilibrium control is thoroughly investigated in this paper. It is shown that the existence of an open-loop time-consistent equilibrium control for a fixed initial pair is equivalent to the solvability of a set of forward-backward stochastic difference equations with stationary conditions and convexity conditions. By decoupling the forward-backward stochastic difference equations, the existence of the open-loop equilibrium control for all the initial pairs is shown to be equivalent to the solvability of a set of coupled constraint generalized difference Riccati equations and a set of coupled constraint linear difference equations.
Equilibrium Controls in Time Inconsistent Stochastic Linear Quadratic Problems

Tianxiao Wang

DOI: https://doi.org/10.1007/s00245-018-9513-x

2018-07-26

Abstract:This paper deals with a class of time inconsistent stochastic linear quadratic optimal control problems in Markovian framework. Three notions, i.e., closed-loop equilibrium strategies, open-loop equilibrium controls and open-loop equilibrium strategies, are characterized in unified manners. These results indicate clearer and deeper distinctions among these notions. For example, in particular time consistent setting, the open-loop equilibrium controls are fully characterized by first-order, second-ordernecessaryoptimalityconditions, and are not optimal in general, while the closed-loop equilibrium controls naturally reduce into closed-loopoptimalcontrols.

mathematics, applied
A Stochastic Optimal Time-Delay Control for Nonlinear Structural Systems

Z. G. Ying,W. Q. Zhu

DOI: https://doi.org/10.12989/sem.2009.31.5.621

2009-01-01

Abstract:The time delay in active and semi-active controls is an important research subject. Many researches on the time-delay control for deterministic systems have been made (Hu and Wang 2002, Yang el al. 1990, Abdel-Mooty and Roorda 1991, Pu 1998, Cai and Huang 2002), while the study on that for stochastic systems is very limited. The effects of the time delay on the control of nonlinear systems under Gaussian white noise excitations have been studied by Bilello et al. (2002). The controlled linear systems with deterministic and random time delay subjected to Gaussian white noise excitations have been treated by Grigoriu (1997). Recently, a stochastic averaging method for quasi-integrable Hamiltonian systems with time delay has been proposed (Liu and Zhu 2007). In the present paper, a stochastic optimal time-delay control method for stochastically excited nonlinear structural systems is proposed based on the stochastic, averaging method for quasi Hamiltonian systems with time delay and the stochastic dynamical,, programming principle. An example of stochastically excited and controlled hysteretic column is given to illustrate the proposed control method.
Stochastic Time-Optimal Control Problems

W ZHANG,D ELLIOTT

DOI: https://doi.org/10.1049/ip-d.1988.0059

1988-01-01

IEE Proceedings D Control Theory and Applications

Abstract:Two types of stochastic time-optimal controls in a one-dimensional setting are considered. Multidimensional problems, in the case of complete state information available and the system modelled by stochastic differential equations, are studied under the formulation of minimising the expected transient-response time. The necessary condition of optimality is the satisfaction for the value function of a parabolic partial differential equation with boundary conditions. The sufficient condition of optimality is also provided, based on Dynkin's formula. Finally, three examples are given
ADVANCES IN RESEARCH ON NONLINEAR STOCHASTIC OPTIMAL CONTROL OF QUASI-HAMILTONIAN SYSTEMS

朱位秋,应祖光

DOI: https://doi.org/10.6052/1000-0992-12-045

2013-01-01

Abstract:Advances in the theory of nonlinear stochastic optimal control of quasi-Hamiltonian systems and its applications in the last decade is summarized. It includes: (1) the basic nonlinear stochastic optimal control strategies based on the stochastic averaging method for quasi-Hamiltonian systems and the stochastic dynamical programming principle, that is, response minimization control, stochastic stabilization, first-passage failure minimization control, and the control for targeting a specified stationary probability density function; (2) further study on the nonlinear stochastic optimal control for application, such as optimal control of partially observable systems, bounded optimal control, time-delay optimal control, semi-active optimal control and minimax control; (3) the integrated nonlinear stochastic optimal control strategy; (4) the applications of the nonlinear stochastic optimal control strategy to engineering structures including hysteretic systems and fractional derivative systems. Some related background and problems to be studied in future are also pointed out.
Stochastic optimal control — A concise introduction

Jiongmin Yong

DOI: https://doi.org/10.3934/mcrf.2020027

2020-01-01

Mathematical Control and Related Fields

Abstract:<p>This is a concise introduction to stochastic optimal control theory. We assume that the readers have basic knowledge of real analysis, functional analysis, elementary probability, ordinary differential equations and partial differential equations. We will present the following topics: (ⅰ) A brief presentation of relevant results on stochastic analysis; (ⅱ) Formulation of stochastic optimal control problems; (ⅲ) Variational method and Pontryagin's maximum principle, together with a brief introduction of backward stochastic differential equations; (ⅳ) Dynamic programming method and viscosity solutions to Hamilton-Jacobi-Bellman equation; (ⅴ) Linear-quadratic optimal control problems, including a careful discussion on open-loop optimal controls and closed-loop optimal strategies, linear forward-backward stochastic differential equations, and Riccati equations.</p>

mathematics, applied
Stochastic Linear-Quadratic Optimal Control Without Time-Consistency Requirement

Yuan-Hua Ni,Ji-Feng Zhang

DOI: https://doi.org/10.4310/cis.2015.v15.n4.a5

2015-01-01

Communications in Information and Systems

Abstract:In this paper, linear-quadratic optimal control without timeconsistency requirement is studied for a class of linear discretetime systems with multiplicative stochastic disturbances. Both the open-loop and the closed-loop time-consistent solutions are investigated. Necessary and sufficient conditions on the existence of the open-loop time-consistent equilibrium control and the closed-loop time-consistent equilibrium strategy are obtained, respectively. Specifically, the existence of the open-loop time-consistent equilibrium control for all the initial time-state pairs is equivalent to the solvability of two coupled constrained linear difference equations and two coupled constrained generalized difference Riccati equations; the existence of the closed-loop time-consistent equilibrium strategy is equivalent to the solvability of another two coupled constrained generalized difference Riccati equations. It can be found that Riccati equations for the open-loop formulation do not admit symmetry structure, while the ones for the closed-loop formulation do have symmetric solutions.
Time Inconsistency and Self-Control Optimization Problems: Progress and Challenges

Yun Shi,Xiangyu Cui

DOI: https://doi.org/10.1007/978-3-319-53518-0_3

2017-01-01

Abstract:Time inconsistency has been an important issue in many stochastic decision problems arisen in real life and financial decision making, especially in the dynamic investment area. When a stochastic decision problem is time inconsistent, the decision maker would be puzzled by his/her conflicting decisions "optimally" derived from his/her time-varying preferences at different time instants. In the literature, the time inconsistent problem is also called the self-control problem, as the decision maker needs to exert proper self-control to resist present temptation and then achieve a better long-term performance. Different approaches dealing with time inconsistency in the literature are reviewed in this paper. After that, the open questions and challenges are also discussed.
On Stochastic Optimal Control of Partially Observable Nonlinear Quasi Hamiltonian Systems

Wei-qui Zhu,Zu-guang Ying

DOI: https://doi.org/10.1007/s11433-010-0081-6

IF: 4.7

2007-01-01

Journal of Sound and Vibration

Abstract:A stochastic optimal control strategy for partially observable nonlinear quasi Hamiltonian systems is proposed. The optimal control forces consist of two parts. The first part is determined by the conditions under which the stochastic optimal control problem of a partially observable nonlinear system is converted into that of a completely observable linear system. The second part is determined by solving the dynamical programming equation derived by applying the stochastic averaging method and stochastic dynamical programming principle to the completely observable linear control system. The response of the optimally controlled quasi Hamiltonian system is predicted by solving the averaged Fok-ker-Planck-Kolmogorov equation associated with the optimally controlled completely observable linear system and solving the Riccati equation for the estimated error of system states. An example is given to illustrate the procedure and effectiveness of the proposed control strategy.
A theory of Markovian time-inconsistent stochastic control in discrete time

Tomas Björk,Agatha Murgoci

DOI: https://doi.org/10.1007/s00780-014-0234-y

IF: 2.0952

2014-06-13

Finance and Stochastics

Abstract:We develop a theory for a general class of discrete-time stochastic control problems that, in various ways, are time-inconsistent in the sense that they do not admit a Bellman optimality principle. We attack these problems by viewing them within a game theoretic framework, and we look for subgame perfect Nash equilibrium points. For a general controlled Markov process and a fairly general objective functional, we derive an extension of the standard Bellman equation, in the form of a system of nonlinear equations, for the determination of the equilibrium strategy as well as the equilibrium value function. Most known examples of time-inconsistent stochastic control problems in the literature are easily seen to be special cases of the present theory. We also prove that for every time-inconsistent problem, there exists an associated time-consistent problem such that the optimal control and the optimal value function for the consistent problem coincide with the equilibrium control and value function, respectively for the time-inconsistent problem. To exemplify the theory, we study some concrete examples, such as hyperbolic discounting and mean–variance control.

social sciences, mathematical methods,mathematics, interdisciplinary applications,statistics & probability,business, finance
STOCHASTIC OPTIMAL CONTROL OF UNCERTAIN QUASI-HAMILTONIAN SYSTEMS

Rongchun Hu,Zuguang Ying,Weiqiu Zhu

DOI: https://doi.org/10.6052/1672-6553-2016-065

2017-01-01

Abstract:In this paper,a stochastic minimax optimal control strategy for uncertain quasi-Hamilionian systems is proposed based on stochastic averaging method,stochastic maximum principle and stochastic differential game theory.Firstly,the partially averaged It(o) stochastic differential equations are derived using the stochastic averaging method for quasi-Hamiltonian systems,while the system state transits from rapid variable of velocity and displacement into the slow variable of energy.Secondly,the stochastic optimal control of Hamiltonian system with a given performance index is converted into a minimax control problem based on the stochastic differential game theory.Thirdly,forward-backward stochastic differential equations of the system and the adjoint process were established according to stochasticmaximum principle.The worst disturbances are generated by minimizing the Hamiltonian function,while maximizing the minimal Hamiltonian function results in the worst-case optimal controls.The worst disturbances and the worst-case optimal controls are then substituting into the partially averaged It(o) equation in order to obtain the fully averaged It(o) equation.The responses of controlled system are predicted by solving the Fokker-Planck-Kolmogorov (FPK) equation associated with the fully averaged It(o) equation.Meanwhile,the control effectiveness can also be computed.Finally,the proposed stochastic optimal control of uncertain quasi-Hamiltonian system is applied into a two-DOF nonlinear system.The effectiveness of the minimax control strategy is validated by numerical results.
Partially Observable Stochastic Optimal Control

Guangchen Wang,Jie Xiong,Shuaiqi Zhang

2016-01-01

International Journal of Numerical Analysis and Modeling

Abstract:This paper is survey on some recent results in optimal control and stochastic filtering. The goal is not to cover all recent developments in control and filtering, instead we focus on maximum principle for optimality of partial information backward or forward-backward stochastic differential equations and branching particle approximation of nonlinear filtering.
On the Time-Inconsistent Deterministic Linear-Quadratic Control

Hongyan Cai,Danhong Chen,Yunfei Peng,Wei

DOI: https://doi.org/10.1137/21m1419611

2022-01-01

Abstract:A fundamental theory of deterministic linear-quadratic (LQ) control is the equivalent relationship between control problems, two-point boundary value problems and Riccati equations. In this paper, we extend the equivalence to a general time-inconsistent deterministic LQ problem, where the inconsistency arises from non-exponential discount functions. By studying the solvability of the Riccati equation, we show the existence and uniqueness of the linear equilibrium for the time-inconsistent LQ problem.
Stochastic Minimax Optimal Time-Delay State Feedback Control of Uncertain Quasi-Integrable Hamiltonian Systems

Ju Feng,Zu-Guang Ying,Yong Wang,Wei-Qiu Zhu

DOI: https://doi.org/10.1007/s00707-011-0530-y

IF: 2.645

2011-01-01

Acta Mechanica

Abstract:A stochastic minimax optimal time-delay state feedback control strategy for uncertain quasi-integrable Hamiltonian systems is proposed. First, a stochastic optimal state feedback control problem of uncertain quasi-integrable Hamiltonian system with time delay in feedback control subjected to Gaussian white noise is formulated. Then, the time-delayed state feedback control forces are approximated by the control forces without time delay and the original problem is converted into a stochastic optimal state feedback control problem of an uncertain system without time delay. After that, by following a procedure based on the stochastic averaging method and stochastic differential game, the worst-case disturbances and the optimal controls are obtained from solving a Hamiltonian–Jacobi–Isaacs (HJI) equation. As an example, the stochastic minimax optimal state feedback control of a Duffing oscillator with parametric disturbances and time-delayed feedback control is worked out in detail to illustrate the procedure and effectiveness of the proposed control strategy.
Time-Inconsistent Mean-Field Stochastic LQ Problem: Open-Loop Time-Consistent Control

Yuan-Hua Ni,Ji-Feng Zhang,Miroslav Krstic

DOI: https://doi.org/10.1109/tac.2017.2776740

IF: 6.549

2017-01-01

IEEE Transactions on Automatic Control

Abstract:This paper is concerned with the open-loop time-consistent solution of time-inconsistent mean-field stochastic linear-quadratic (LQ) optimal control. Different from standard stochastic linear-quadratic problems, both the system matrices and the weighting matrices are depending on the initial times, and the conditional expectations of the control and state enter quadratically into the cost functional. Such features will ruin Bellman's principle of optimality and result in the time inconsistency of optimal control. Based on the dynamical nature of the systems involved, a kind of open-loop time-consistent equilibrium control is investigated in this paper. It is shown that the existence of open-loop equilibrium control for a fixed initial pair is equivalent to the solvability of a set of forward-backward stochastic difference equations with stationary condition and convexity condition. By decoupling the forward-backward stochastic difference equations, necessary and sufficient conditions in terms of linear difference equations and generalized difference Riccati equations are given for the existence of open-loop equilibrium control for a fixed initial pair. Moreover, the existence of open-loop time-consistent equilibrium controls for all the initial pairs is shown to be equivalent to the solvability of a set of coupled constrained generalized difference Riccati equations and two sets of constrained linear difference equations.
Optimal Timing Control of Discrete-Time Linear Switched Stochastic Systems

Xiaomei Liu,Kanjian Zhang,Shengtao Li,Shumin Fei,Haikun Wei

DOI: https://doi.org/10.1007/s12555-013-0261-z

IF: 2.964

2014-01-01

International Journal of Control Automation and Systems

Abstract:Optimal switch-time control is the study that investigates how best to switch between different modes. In this paper, we investigate the optimal switch-time control problem for discrete-time linear switched stochastic systems. In particular, under the assumption that the sequence of active subsystems is pre-specified, we focus on the problem where the objective is to minimize a cost functional defined on the states and the switching times are the only control variables. For systems with one switching time, using calculus of variations, we firstly derive the difference formulae of the cost functional with respect to the switching time, which can be directly used to find the optimal switching instant. Then, a method is presented to deal with the problem with multiple switching times case. Finally, the viability of the proposed method is illustrated through two numerical examples.

An Introductory Review of Time-Inconsistent Stochastic Optimal Control

Time-Inconsistent Discrete-Time Stochastic Linear-Quadratic Optimal Control : Time-consistent Solutions

Discrete-Time Stochastic Linear-Quadratic Optimal Control with Time-Inconsistency

On Time-Consistent Solution to Time-Inconsistent Linear-Quadratic Optimal Control of Discrete-Time Stochastic Systems

Stochastic Fractional Optimal Control of Quasi-Integrable Hamiltonian System with Fractional Derivative Damping

Time-inconsistent Mean-Field Stochastic Linear-Quadratic Optimal Control

Equilibrium Controls in Time Inconsistent Stochastic Linear Quadratic Problems

A Stochastic Optimal Time-Delay Control for Nonlinear Structural Systems

Stochastic Time-Optimal Control Problems

ADVANCES IN RESEARCH ON NONLINEAR STOCHASTIC OPTIMAL CONTROL OF QUASI-HAMILTONIAN SYSTEMS

Stochastic optimal control — A concise introduction

Stochastic Linear-Quadratic Optimal Control Without Time-Consistency Requirement

Time Inconsistency and Self-Control Optimization Problems: Progress and Challenges

On Stochastic Optimal Control of Partially Observable Nonlinear Quasi Hamiltonian Systems

A theory of Markovian time-inconsistent stochastic control in discrete time

STOCHASTIC OPTIMAL CONTROL OF UNCERTAIN QUASI-HAMILTONIAN SYSTEMS

Partially Observable Stochastic Optimal Control

On the Time-Inconsistent Deterministic Linear-Quadratic Control

Stochastic Minimax Optimal Time-Delay State Feedback Control of Uncertain Quasi-Integrable Hamiltonian Systems

Time-Inconsistent Mean-Field Stochastic LQ Problem: Open-Loop Time-Consistent Control

Optimal Timing Control of Discrete-Time Linear Switched Stochastic Systems