Abstract:Among various rare events, the effective computation of transition paths connecting metastable states in a stochastic model is an important problem. This paper proposes a stochastic optimal control formulation for transition path problems in an infinite time horizon for Markov jump processes on polish space. An unbounded terminal cost at a stopping time and a controlled transition rate for the jump process regulate the transition from one metastable state to another. The running cost is taken as an entropy form of the control velocity, in contrast to the quadratic form for diffusion processes. Using the Girsanov transformation for Markov jump processes, the optimal control problem in both finite time and infinite time horizon with stopping time fit into one framework: the optimal change of measures in the Càdlàg path space via minimizing their relative entropy. We prove that the committor function, solved from the backward equation with appropriate boundary conditions, yields an explicit formula for the optimal path measure and the associated optimal control for the transition path problem. The unbounded terminal cost leads to a singular transition rate (unbounded control velocity), for which, the Gamma convergence technique is applied to pass the limit for a regularized optimal path measure. The limiting path measure is proved to solve a Martingale problem with an optimally controlled transition rate and the associated optimal control is given by Doob-h transformation. The resulting optimally controlled process can realize the transitions almost surely.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to effectively calculate the transition paths connecting different metastable states in Markov Jump Processes (MJPs). Specifically, the author proposes a stochastic optimal control framework for handling this transition path problem in an infinite - time horizon. This method regulates the transition process from one metastable state to another by introducing an unbounded terminal cost and a controlled transition rate. The running cost adopts the entropy form of the control speed instead of the quadratic form in the diffusion process. Using the Girsanov transformation of Markov jump processes, the optimal control problems in both finite - time and infinite - time horizons can be solved under a unified framework: that is, by minimizing their relative entropy, the change of the optimal measure is achieved in the Càdlàg path space. The paper also proves that the flux function (commitment function) obtained by solving the backward equation with appropriate boundary conditions can give explicit formulas for the optimal path measure of the transition path problem and its associated optimal control. For the singular transition rate caused by the unbounded terminal cost, the Gamma - convergence technique is applied to deal with the limit problem of the regularized optimal path measure, and it is proved that the limit path measure can solve a martingale problem with an optimal - control transition rate, and the corresponding optimal control can be given by the Doob - h transformation. Ultimately, the proposed optimal control process can achieve the transition almost surely. In short, the core problem of the paper is to develop an effective mathematical framework for calculating and understanding rare events occurring in Markov jump processes, especially the transition paths between two metastable states. This framework not only provides a theoretical explanation but also a method for the practical calculation of these transition paths.

Optimal control formulation of transition path problems for Markov Jump Processes

Optimal Vibration Control of a Class of Nonlinear Stochastic Systems with Markovian Jump

An Optimal Control Method to Compute the Most Likely Transition Path for Stochastic Dynamical Systems with Jumps

H∞$$ {h}_{\infty } $$ Optimal Output Tracking Control for Markov Jump Systems: A Reinforcement Learning‐based Approach

Optimal Stopping Problem for Jump–diffusion Processes with Regime-Switching

A Stochastic Linear-Quadratic Optimal Control Problem with Jumps in an Infinite Horizon

TIME-INCONSISTENT OPTIMAL CONTROL PROBLEMS WITH REGIME-SWITCHING

Optimal control of mode transition rates/probabilities for Markovian jump systems

From the Optimal Singular Stochastic Control to the Optimal Stopping for Regime-Switching Processes.

Finite-Time Control for Discrete-Time Positive Systems Subject to Event-Triggered Scheme and Markov Jump Parameters

Stochastic Control Problems for Systems Driven by Normal Martingales

H2/H∞ Control for Stochastic Jump-Diffusion Systems with Markovian Switching

Mean-field Stochastic Linear–quadratic Optimal Control with Markov Jump Parameters

Finite-time optimal control for Markov jump systems with singular perturbation and hard constraints

Sufficient Stochastic Maximum Principle for the Optimal Control of Semi-Markov Modulated Jump-Diffusion with Application to Financial Optimization

Near-Optimality in Stochastic Control of an Capital Accumulation System with Markovian Switching and Poisson Jumps

Optimal Control and Stabilization Problem for Discrete-time Markov Jump Systems with Indefinite Weight Costs

Continuous-Time Controlled Markov Chains

Stochastic Near-Optimal Controls for Path-Dependent Systems

Optimal Stationary Policies for a Class of Countable Markov Control Processes

Necessary Conditions for Optimal Control of Stochastic Systems with Random Jumps