The optimal control related to Riemannian manifolds and the viscosity solutions to H-J-B equations

Xuehong Zhu

DOI: https://doi.org/10.48550/arXiv.1001.2820

2010-01-16

Probability

Abstract:This paper is concerned with the Dynamic Programming Principle (DPP in short) with SDEs on Riemannian manifolds. Moreover, through the DPP, we conclude that the cost function is the unique viscosity solution to the related PDEs on manifolds.

What problem does this paper attempt to address?

Optimal control problems with generalized mean-field dynamics and viscosity solution to Master Bellman equation

Rainer Buckdahn,Juan Li,Zhanxin Li

2024-08-15

Abstract:We study an optimal control problem of generalized mean-field dynamics with open-loop controls, where the coefficients depend not only on the state processes and controls, but also on the joint law of them. The value function $V$ defined in a conventional way, but it does not satisfy the Dynamic Programming Principle (DPP for short). For this reason we introduce subtly a novel value function $\vartheta$, which is closely related to the original value function $V$, such that, a description of $\vartheta$, as a solution of a partial differential equation (PDE), also characterizes $V$. We establish the DPP for $\vartheta$. By using an intrinsic notion of viscosity solutions, initially introduced in Burzoni, Ignazio, Reppen and Soner [8] and specifically tailored to our framework, we show that the value function $\vartheta$ is a viscosity solution to a Master Bellman equation on a subset of Wasserstein space of probability measures. The uniqueness of viscosity solution is proved for coefficients which depend on the time and the joint law of the control process and the controlled process. Our approach is inspired by Buckdahn, Li, Peng and Rainer [7], and leads to a generalization of the mean-field PDE in [7] to a Master Bellman equation in the case of controls.

Optimization and Control,Probability
Optimal control problem for reflected McKean-Vlasov SDEs

Jinghai Shao

2023-09-15

Abstract:This work investigates the optimal control problem for reflected McKean-Vlasov SDEs and the viscosity solutions to Hamilton-Jacobi-Bellman(HJB) equations on the Wasserstein space in terms of intrinsic derivative. It follows from the flow property of reflected McKean-Vlasov SDEs that the dynamic programming principle holds. Applying the decoupling method and the heat kernel estimates for parabolic equations, we show that the value function is a viscosity solution to an appropriate HJB equation on the Wasserstein space, where the characterization of absolutely continuous curves on the Wasserstein space by the continuity equations plays an important role. To establish the uniqueness of viscosity solution, we generalize the construction of a distance like function initiated in Burzoni et al.(SICON, 2020) to the Wasserstein space over multidimensional space and show its effectiveness to cope with HJB equations in terms of intrinsic derivative on the Wasserstein space.

Probability,Analysis of PDEs,Optimization and Control
Connection Between MP and DPP for Stochastic Recursive Optimal Control Problems: Viscosity Solution Framework in Local Case.

Tianyang Nie,Jingtao Shi,Zhen Wu

DOI: https://doi.org/10.1109/acc.2016.7526813

2016-01-01

Abstract:This paper deals with a nonsmooth version of the connection between the maximum principle and dynamic programming principle, for the stochastic recursive control problem when the control domain is convex. By employing the notions of sub- and super-jets, the set inclusions are derived among the value function and the adjoint processes. The general case for non-convex control domain is open.
A Survey of Optimal Control Problems Evolved on Riemannian Manifolds

Li Deng,Xu Zhang

DOI: https://doi.org/10.4208/csiam-am.so-2021-0018

2022-01-01

CSIAM Transactions on Applied Mathematics

Abstract:In this paper, we present our optimality results on optimal control problems for ordinary differential equations on Riemannian manifolds. For the problems with free states at the terminal time, we obtain the first and second-order necessary conditions, dynamical programming principle, and their relations. Then, we consider the problems with the initial and final states satisfying some inequality-type and equalitytype constraints, and establish the corresponding first and second-order necessary conditions of optimal pairs in the sense of either spike or convex variations. For each of the above results concerning second-order optimality conditions, the curvature tensor of the underlying manifold plays a crucial role.
Viscosity Solutions to HJB Equations Associated with Optimal Control Problem for McKean-Vlasov SDEs

Jinghai Shao

DOI: https://doi.org/10.48550/arxiv.2310.11474

2023-01-01

Abstract:This work concerns the optimal control problem for McKean-Vlasov SDEs. In order to characterize the value function, we develop the viscosity solution theory for Hamilton-Jacobi-Bellman (HJB) equations on the Wasserstein space using Mortensen's derivative. In particular, a comparison principle for viscosity solution is established. Our approach is based on Borwein-Preiss variational principle to overcome the loss of compactness for bounded sets in the Wasserstein space.
Path-Dependent Optimal Stochastic Control and Viscosity Solution of Associated Bellman Equations

Shanjian Tang,Fu Zhang

DOI: https://doi.org/10.3934/dcds.2015.35.5521

2015-01-01

Discrete and Continuous Dynamical Systems

Abstract:In this paper we study the optimal stochastic control problem for a path-dependent stochastic system under a recursive path-dependent cost functional, whose associated Bellman equation from dynamic programming principle is a path-dependent fully nonlinear partial differential equation of second order. A novel notion of viscosity solutions is introduced. Using Dupire's functional Itô calculus, we characterize the value functional of the optimal stochastic control problem as the unique viscosity solution to the associated path-dependent Bellman equation.
Existence of viscosity solutions for Hamilton-Jacobi equations on Riemannian Manifolds via Lyapunov control

Serena Della Corte,Richard C. Kraaij

2024-07-17

Abstract:We give a new perspective on the existence of viscosity solutions for a stationary and a time-dependent first order Hamilton-Jacobi equation. Following recent comparison principles, we work in a framework in which we consider a subsolution and a supersolution for two equations in terms of two Hamiltonians that can be seen as an upper and lower bound of our original Hamiltonian respectively. The upper and lower bound are formulated in terms of a relaxed Lyapunov function which allows us to restrict part of the analysis to compact sets and to work with almost optimizers of the considered control problems. For this reason, we can relax assumptions on the control problem: most notably, we do not need completeness of set of controlled paths. Moreover, our strategy avoids a-priori analysis on the regularity of the candidate solutions. We then explore the context of Hamilton-Jacobi-Bellman and Hamilton-Jacobi-Isaacs equations to verify our assumptions in a convex and non-convex setting respectively.

Analysis of PDEs
Relationship between General MP and DPP for the Stochastic Recursive Optimal Control Problem With Jumps: Viscosity Solution Framework

Bin Wang,Jingtao Shi

2024-06-03

Abstract:This paper is concerned with the relationship between general maximum principle and dynamic programming principle for the stochastic recursive optimal control problem with jumps, where the control domain is not necessarily convex. Relations among the adjoint processes, the generalized Hamiltonian function and the value function are proved, under the assumption of a smooth value function and within the framework of viscosity solutions, respectively. Some examples are given to illustrate the theoretical results.

Optimization and Control
Deterministic Optimal Control on Riemannian Manifolds Under Probability Knowledge of the Initial Condition

Frédéric Jean,Othmane Jerhaoui,Hasnaa Zidani

DOI: https://doi.org/10.1137/23m1575251

IF: 2.071

2024-05-04

SIAM Journal on Mathematical Analysis

Abstract:SIAM Journal on Mathematical Analysis, Volume 56, Issue 3, Page 3326-3356, June 2024. In this article, we study a Mayer optimal control problem on the space of Borel probability measures over a compact Riemannian manifold [math]. This is motivated by certain situations where a central planner of a deterministic controlled system has only imperfect information on the initial state of the system. The lack of information here is very specific. It is described by a Borel probability measure along which the initial state is distributed. We define a new notion of viscosity in this space by taking test functions that are directionally differentiable and can be written as a difference of two semiconvex functions. With this choice of test functions, we extend the notion of viscosity to Hamilton–Jacobi–Bellman equations in Wasserstein spaces and we establish that the value function is the unique viscosity solution of a Hamilton–Jacobi–Bellman equation in the Wasserstein space over [math].

mathematics, applied
Stochastic maximum principle, dynamic programming principle, and their relationship for fully coupled forward-backward stochastic controlled systems

Mingshang Hu,Shaolin Ji,Xiaole Xue

DOI: https://doi.org/10.1051/cocv/2020051

2020-01-01

Abstract:Within the framework of viscosity solution, we study the relationship between the maximum principle (MP) from M. Hu, S. Ji and X. Xue [ SIAM J. Control Optim. 56 (2018) 4309–4335] and the dynamic programming principle (DPP) from M. Hu, S. Ji and X. Xue [ SIAM J. Control Optim. 57 (2019) 3911–3938] for a fully coupled forward–backward stochastic controlled system (FBSCS) with a nonconvex control domain. For a fully coupled FBSCS, both the corresponding MP and the corresponding Hamilton–Jacobi–Bellman (HJB) equation combine an algebra equation respectively. With the help of a new decoupling technique, we obtain the desirable estimates for the fully coupled forward–backward variational equations and establish the relationship. Furthermore, for the smooth case, we discover the connection between the derivatives of the solution to the algebra equation and some terms in the first-order and second-order adjoint equations. Finally, we study the local case under the monotonicity conditions as from J. Li and Q. Wei [ SIAM J. Control Optim. 52 (2014) 1622–1662] and Z. Wu [ Syst. Sci. Math. Sci. 11 (1998) 249–259], and obtain the relationship between the MP from Z. Wu [ Syst. Sci. Math. Sci. 11 (1998) 249–259] and the DPP from J. Li and Q. Wei [ SIAM J. Control Optim. 52 (2014) 1622–1662].
Stochastic Recursive Optimal Control of McKean-Vlasov Type: A Viscosity Solution Approach

Liangquan Zhang

2024-07-08

Abstract:In this paper, we study a kind of optimal control problem for forward-backward stochastic differential equations (FBSDEs for short) of McKean--Vlasov type via the dynamic programming principle (DPP for short) motivated by studying the infinite dimensional Hamilton--Jacobi--Bellman (HJB for short) equation derived from the decoupling field of the FBSDEs posed by Carmona and Delarue (\emph{Ann Probab}, 2015, \cite{cd15}). At the beginning, the value function is defined by the solution to the controlled BSDE alluded to earlier. On one hand, we can prove the value function is deterministic function with respect to the initial random variable; On the other hand, we can show that the value function is \emph{law-invariant}, i.e., depending on only via its distribution by virtue of BSDE property. Afterward, utilizing the notion of differentiability with respect to probability measures introduced by P.L. Lions \cite{Lions2012}, we are able to establish a DPP for the value function in the Wasserstein space of probability measures based on the application of BSDE approach, particularly, employing the notion of stochastic \emph{backward semigroups} associated with stochastic optimal control problems and Itô formula along a flow property of the conditional law of the controlled forward state process. We prove that the value function is the unique viscosity solutions of the associated generalized HJB equations in some separable Hilbert space. Finally, as an application, we formulate an optimal control problem for linear stochastic differential equations with quadratic cost functionals of McKean-Vlasov type under nonlinear expectation, $g$-expectation introduced by Peng \cite{Peng04} and derive the optimal feedback control explicitly by means of several groups of Riccati equations.

Optimization and Control
Stochastic maximum principle, dynamic programming principle, and their relationship for fully coupled forward-backward stochastic control systems

Mingshang Hu,Shaolin Ji,Xiaole Xue

DOI: https://doi.org/10.48550/arXiv.1803.06804

2018-05-16

Abstract:Within the framework of viscosity solution, we study the relationship between the maximum principle (MP) in [9] and the dynamic programming principle (DPP) in [10] for a fully coupled forward-backward stochastic controlled system (FBSCS) with a nonconvex control domain. For a fully coupled FBSCS, both the corresponding MP and the corresponding Hamilton-Jacobi-Bellman (HJB) equation combine an algebra equation respectively. So this relationship becomes more complicated and almost no work involves this issue. With the help of a new decoupling technique, we obtain the desirable estimates for the fully coupled forward-backward variational equations and establish the relationship. Furthermore, for the smooth case, we discover the connection between the derivatives of the solution to the algebra equation and some terms in the first and second-order adjoint equations. Finally, we study the local case under the monotonicity conditions as in [14,27] and obtain the relationship between the MP in [27] and the DPP in [14].

Optimization and Control
THE EXISTENCE AND UNIQUENESS OF VISCOSITY SOLUTION TO A KIND OF HAMILTON-JACOBI-BELLMAN EQUATION

Mingshang Hu,Shaolin Ji,Xiaole Xue

DOI: https://doi.org/10.1137/18m1231833

IF: 2.2

2019-01-01

SIAM Journal on Control and Optimization

Abstract:In this paper, we study the existence and uniqueness of viscosity solutions to a kind of Hamilton-Jacobi-Bellman (HJB) equations combined with algebra equations. This HJB equation is related to a stochastic optimal control problem for which the state equation is described by a fully coupled forward-backward stochastic differential equation. By extending Peng's backward semigroup approach to this problem, we obtain the dynamic programming principle and show that the value function is a viscosity solution to this HJB equation. As for the proof of the uniqueness of viscosity solution, the analysis method in Barles, Buckdahn and Pardoux Baeles-BP usually does not work for this fully coupled case. With the help of the uniqueness of the solution to FBSDEs, we propose a novel probabilistic approach to study the uniqueness of the solution to this HJB equation. We obtain that the value function is the minimum viscosity solution to this HJB equation. Especially, when the coefficients are independent of the control variable or the solution is smooth, the value function is the unique viscosity solution.
Viscosity Solutions Approach to Finite-Horizon Continuous-Time Markov Decision Process

Zhong-Wei Liao,Jinghai Shao

DOI: https://doi.org/10.1080/00207179.2024.2338448

IF: 2.102

2024-01-01

International Journal of Control

Abstract:This paper investigates the optimal control problems for the finite-horizon continuous-time Markov decision processes with delay-dependent control policies. We develop compactification methods in decision processes and show that the existence of optimal policies. Subsequently, through the dynamic programming principle of the delay-dependent control policies, the differential-difference Hamilton-Jacobi-Bellman (HJB) equation in the setting of discrete space is established. Under certain conditions, we give the comparison principle and further prove that the value function is the unique viscosity solution to this HJB equation. Based on this, we show that among the class of delay-dependent control policies, there is an optimal one which is Markovian.
Optimal Control of the Viscous Degasperis-Procesi Equation

Lixin Tian,Chunyu Shen

DOI: https://doi.org/10.1063/1.2804755

IF: 1.469

2007-01-01

Journal of Mathematical Physics

Abstract:This paper studies the problem for optimal control of the viscous Degasperis-Procesi equation. The existence and uniqueness of weak solution to the equation are proved in a short interval. The optimal control of the viscous Degasperis-Procesi equation under boundary condition is given and the existence of optimal solution to the equation is proved.
Optimal control problems of fully coupled FBSDEs and viscosity solutions of Hamilton-Jacobi-Bellman equations

Juan Li,Qingmeng Wei

DOI: https://doi.org/10.48550/arXiv.1302.0935

2013-02-05

Abstract:In this paper we study stochastic optimal control problems of fully coupled forward-backward stochastic differential equations (FBSDEs). The recursive cost functionals are defined by controlled fully coupled FBSDEs. We study two cases of diffusion coefficients $\sigma$ of FSDEs. We use a new method to prove that the value functions are deterministic, satisfy the dynamic programming principle (DPP), and are viscosity solutions to the associated generalized Hamilton-Jacobi-Bellman (HJB) equations. The associated generalized HJB equations are related with algebraic equations when $\sigma$ depends on the second component of the solution $(Y, Z)$ of the BSDE and doesn't depend on the control. For this we adopt Peng's BSDE method, and so in particular, the notion of stochastic backward semigroup in [16]. We emphasize that the fact that $\sigma$ also depends on $Z$ makes the stochastic control much more complicate and has as consequence that the associated HJB equation is combined with an algebraic equation, which is inspired by Wu and Yu [19]. We use the continuation method combined with the fixed point theorem to prove that the algebraic equation has a unique solution, and moreover, we also give the representation for this solution. On the other hand, we prove some new basic estimates for fully coupled FBSDEs under the monotonic assumptions. In particular, we prove under the Lipschitz and linear growth conditions that fully coupled FBSDEs have a unique solution on the small time interval, if the Lipschitz constant of $\sigma$\ with respect to $z$ is sufficiently small. We also establish a generalized comparison theorem for such fully coupled FBSDEs.

Optimization and Control,Probability
Maximum principle for viscosity solutions on Riemannian manifolds

Shige Peng,Detang Zhou

DOI: https://doi.org/10.48550/arXiv.0806.4768

2008-06-29

Differential Geometry

Abstract:In this work we consider viscosity solutions to second order partial differential equations on Riemannian manifolds. We prove maximum principles for solutions to Dirichlet problem on a compact Riemannian manifold with boundary. Using a different method, we generalize maximum principles of Omori and Yau to a viscosity version.
Necessary and Sufficient Conditions for Optimal Control of Semilinear Stochastic Partial Differential Equations

Wilhelm Stannat,Lukas Wessels

DOI: https://doi.org/10.1214/23-AAP2038

2023-12-08

Abstract:Using a recently introduced representation of the second order adjoint state as the solution of a function-valued backward stochastic partial differential equation (SPDE), we calculate the viscosity super- and subdifferential of the value function evaluated along an optimal trajectory for controlled semilinear SPDEs. This establishes the well-known connection between Pontryagin's maximum principle and dynamic programming within the framework of viscosity solutions. As a corollary, we derive that the correction term in the stochastic Hamiltonian arising in non-smooth stochastic control problems is non-positive. These results directly lead us to a stochastic verification theorem for fully nonlinear Hamilton--Jacobi--Bellman equations in the framework of viscosity solutions.

Probability,Optimization and Control
Linear Quadratic Optimal Control for Systems Governed by First-Order Hyperbolic Partial Differential Equations

Xiaomin Xue,Juanjuan Xu,Huanshui Zhang

DOI: https://doi.org/10.1007/s11424-024-3324-8

2024-02-29

Journal of Systems Science and Complexity

Abstract:This paper focuses on linear-quadratic (LQ) optimal control for a class of systems governed by first-order hyperbolic partial differential equations (PDEs). Different from most of the previous works, an approach of discretization-then-continuousization is proposed in this paper to cope with the infinite-dimensional nature of PDE systems. The contributions of this paper consist of the following aspects: 1) The differential Riccati equations and the solvability condition of the LQ optimal control problems are obtained via the discretization-then-continuousization method. 2) A numerical calculation way of the differential Riccati equations and a practical design way of the optimal controller are proposed. Meanwhile, the relationship between the optimal costate and the optimal state is established by solving a set of forward and backward partial difference equations (FBPDEs). 3) The correctness of the method used in this paper is verified by a complementary continuous method and the comparative analysis with the existing operator results is presented. It is shown that the proposed results not only contain the classic results of the standard LQ control problem of systems governed by ordinary differential equations as a special case, but also support the existing operator results and give a more convenient form of computation.

mathematics, interdisciplinary applications
On Viscosity Solution of Hjb Equations with State Constraints and Reflection Control

Anup Biswas,Hitoshi Ishii,Subhamay Saha,Lin Wang

DOI: https://doi.org/10.1137/15m103830x

IF: 2.2

2017-01-01

SIAM Journal on Control and Optimization

Abstract:Motivated by a control problem of a certain queueing network we consider a control problem where the dynamics is constrained in the nonnegative orthant R-+(d) of the d-dimensional Euclidean space and controlled by the reflections at the faces/boundaries. We define a discounted value function associated to this problem and show that the value function is a viscosity solution to a certain HJB equation in R-+(d) with nonlinear Neumann type boundary condition. Under certain conditions, we also characterize this value function as the unique solution to this HJB equation.

The optimal control related to Riemannian manifolds and the viscosity solutions to H-J-B equations

Optimal control problems with generalized mean-field dynamics and viscosity solution to Master Bellman equation

Optimal control problem for reflected McKean-Vlasov SDEs

Connection Between MP and DPP for Stochastic Recursive Optimal Control Problems: Viscosity Solution Framework in Local Case.

A Survey of Optimal Control Problems Evolved on Riemannian Manifolds

Viscosity Solutions to HJB Equations Associated with Optimal Control Problem for McKean-Vlasov SDEs

Path-Dependent Optimal Stochastic Control and Viscosity Solution of Associated Bellman Equations

Existence of viscosity solutions for Hamilton-Jacobi equations on Riemannian Manifolds via Lyapunov control

Relationship between General MP and DPP for the Stochastic Recursive Optimal Control Problem With Jumps: Viscosity Solution Framework

Deterministic Optimal Control on Riemannian Manifolds Under Probability Knowledge of the Initial Condition

Stochastic maximum principle, dynamic programming principle, and their relationship for fully coupled forward-backward stochastic controlled systems

Stochastic Recursive Optimal Control of McKean-Vlasov Type: A Viscosity Solution Approach

Stochastic maximum principle, dynamic programming principle, and their relationship for fully coupled forward-backward stochastic control systems

THE EXISTENCE AND UNIQUENESS OF VISCOSITY SOLUTION TO A KIND OF HAMILTON-JACOBI-BELLMAN EQUATION

Viscosity Solutions Approach to Finite-Horizon Continuous-Time Markov Decision Process

Optimal Control of the Viscous Degasperis-Procesi Equation

Optimal control problems of fully coupled FBSDEs and viscosity solutions of Hamilton-Jacobi-Bellman equations

Maximum principle for viscosity solutions on Riemannian manifolds

Necessary and Sufficient Conditions for Optimal Control of Semilinear Stochastic Partial Differential Equations

Linear Quadratic Optimal Control for Systems Governed by First-Order Hyperbolic Partial Differential Equations

On Viscosity Solution of Hjb Equations with State Constraints and Reflection Control