Continuous Time Markov Decision Processes with Discounted Moment Criterion

QY Hu

DOI: https://doi.org/10.1006/jmaa.1996.9999

IF: 2.2

1992-01-01

Optimization

Abstract:Time Markov decision processes with countable states and actions continuous are discussed with the criterion of discounted moment optimality. They are transformed into a sequence of discrete time Markov decision processes with the criterion of discounted expected total rewards.

What problem does this paper attempt to address?

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

QY Hu,JY Liu,WY Yue

DOI: https://doi.org/10.1007/3-540-44862-4_8

2003-01-01

Abstract:This paper discusses continuous time Markov decision processes with criterion of expected discounted total rewards, where the state space is countable, the reward rate function is extended real-valued and the discount rate is a real number. Under necessary conditions that the model is well defined, the state space is partitioned into three subsets, on which the optimal value function is positive infinity, negative infinity, or finite, respectively. Correspondingly, the model is reduced into three submodels, by generalizing policies and eliminating some worst actions. Then for the submodel with finite optimal value, the validity of the optimality equation is shown and some its properties are obtained.
Mixed Markov Decision Processes in a Semi-Markov Environment with Discounted Criterion

QY Hu,JL Wang

DOI: https://doi.org/10.1006/jmaa.1997.5792

IF: 1.417

1998-01-01

Journal of Mathematical Analysis and Applications

Abstract:This paper presents a new model: the mixed Markov decision process (MDP) in a semi-Markov environment with discounted criterion. It describes a system which behaves like a MDP except that the system is influenced by its semi-Markov process environment. Following each state transition of the environment, the MDP model changes among discrete time MDP, continuous time MDP, and semi-MDP. After presenting the model, we show the validity of the optimality equation and the existence of ε-optimal policies. Finally, the mixed MDP in a Markov environment is transformed into a discrete time MDP.
Continuous Time Markov Decision Processes with Nonuniformly Bounded Transition Rate: Expected Total Rewards

Qiying. Hu ∗,jinling. Wang

DOI: https://doi.org/10.1080/02331939808844385

IF: 2.2

1998-01-01

Optimization

Abstract:This paper deals with a stationary continuous time Markov decision process with countable state space, nonuniformly bounded transition rate and extended real valued reward, under the criterion of expected total rewards. The model is transformed into an equivalent discrete time Markov decision process within stochastic stationary policies when the model is well-defined. Thus most results, for example, the optimality equation and the existence of e- optimal policies, in the discrete time Markov decision process may be generalized directly to hold for the continuous time Markov decision process. Furthermore, the state space is partitioned into three subsets: where the optimal value equals to positive infinity, negative infinity, or is finite, respectively
Analysis for Some Properties of Discrete Time Markov Decision Processes

QY Hu,WI Yue

DOI: https://doi.org/10.1080/02331930310001611493

IF: 2.2

2003-01-01

Optimization

Abstract:This paper investigates properties of the optimality equation and optimal policies in discrete time Markov decision processes with expected discounted total rewards under weak conditions that the model is well defined and the optimality equation is true. The optimal value function is characterized as a solution of the optimality equation and the structure of optimal policies is also given.
Constrained Markov Decision Processes with Non-constant Discount Factor

Héctor Jasso-Fuentes,Tomás Prieto-Rumeau

DOI: https://doi.org/10.1007/s10957-024-02453-y

2024-06-01

Journal of Optimization Theory and Applications

Abstract:This paper studies constrained Markov decision processes under the total expected discounted cost optimality criterion, with a state-action dependent discount factor that may take any value between zero and one. Both the state and the action space are assumed to be Borel spaces. By using the linear programming approach, consisting in stating the control problem as a linear problem on a set of occupation measures, we show the existence of an optimal stationary Markov policy. Our results are based on the study of both weak-strong topologies in the space of occupation measures and Young measures in the space of Markov policies.

mathematics, applied,operations research & management science
A survey of recent results on continuous-time Markov decision processes

Xianping Guo,Onésimo Hernández-Lerma,Tomás Prieto-Rumeau,Xi-Ren Cao,Junyu Zhang,Qiying Hu,Mark E. Lewis,Ricardo Vélez

DOI: https://doi.org/10.1007/BF02837562

IF: 1.7

2006-01-01

Top

Abstract:This paper is a survey of recent results on continuous-time Markov decision processes (MDPs) with unbounded transition rates, and reward rates that may be unbounded from above and from below. These results pertain to discounted and average reward optimality criteria, which are the most commonly used criteria, and also to more selective concepts, such as bias optimality and sensitive discount criteria. For concreteness, we consider only MDPs with a countable state space, but we indicate how the results can be extended to more general MDPs or to Markov games.
The Finiteness of the Reward Function and the Optimal Value Function in Markov Decision Processes

Qiying Hu,Chen Xu

DOI: https://doi.org/10.1007/pl00020916

1999-01-01

Mathematical Methods of Operations Research

Abstract:. This paper studies the discrete time Markov decision processes (MDP) with expected discounted total reward, where the state space is countable, the action space is measurable, the reward function is extended real-valued, and the discount rate may be any real number. Two conditions (GC) and (C) are presented, which are weaker than that presented in literature. By eliminating some worst actions, the state space S can be partitioned into sets S ∞ , S −∞ , S 0 , on which the optimal value function equals +∞ , −∞ or is finite, respectively. Furthermore, the validity of the optimality equation is shown when its right hand side is well defined, especially, when it is restricted to the subset S 0 . The reward function r ( i , a ) is finite and bounded above in a for each i ∈ S 0 . Finally, some sufficient conditions for (GC) and (C) are given.
Markov Decision Processes with Time-Varying Geometric Discounting

Jiarui Gan,Annika Hennes,Rupak Majumdar,Debmalya Mandal,Goran Radanovic

DOI: https://doi.org/10.1609/aaai.v37i10.26413

2023-07-20

Abstract:Canonical models of Markov decision processes (MDPs) usually consider geometric discounting based on a constant discount factor. While this standard modeling approach has led to many elegant results, some recent studies indicate the necessity of modeling time-varying discounting in certain applications. This paper studies a model of infinite-horizon MDPs with time-varying discount factors. We take a game-theoretic perspective -- whereby each time step is treated as an independent decision maker with their own (fixed) discount factor -- and we study the subgame perfect equilibrium (SPE) of the resulting game as well as the related algorithmic problems. We present a constructive proof of the existence of an SPE and demonstrate the EXPTIME-hardness of computing an SPE. We also turn to the approximate notion of $\epsilon$-SPE and show that an $\epsilon$-SPE exists under milder assumptions. An algorithm is presented to compute an $\epsilon$-SPE, of which an upper bound of the time complexity, as a function of the convergence property of the time-varying discount factor, is provided.

Artificial Intelligence,Computer Science and Game Theory
Performance Optimization for Countable Semi-Markov Decision Processes with Discounted-cost

YIN Bao-qun,LI Yan-jie,ZHOU Ya-ping,XI Hong-sheng

DOI: https://doi.org/10.3321/j.issn:1001-0920.2006.08.020

2006-01-01

Abstract:The problem of discounted-cost performance optimization is addressed for a class of countable semi-Markov decision processes(CSMDPs).By using an equivalent Markov process,the discounted Poisson equation is proposed for a CSMDP.This equation is used to define the α-potential,based on which the optimality equation satisfied by the optimal stationary policy is derived.The existence of solutions to the optimality equation is discussed and some sufficient conditions for the existence of solutions are given.
Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

Baoqun Yin,Yanjie Li,Yaping Zhou,Hongsheng Xi

DOI: https://doi.org/10.3166/ejc.14.213-222

IF: 2.649

2008-01-01

European Journal of Control

Abstract:We discuss the problems of discounted-cost performance optimization for a class of semi-Markov decision processes (SMDPs). We define a matrix which can be used as the infinitesimal generator of a Markov process. The discounted Poisson equation is proposed for an SMDP by using this matrix, from which the α-potential is defined. The optimality equation satisfied by the optimal stationary policy is given and the relation between discounted model and average model is discussed. Two iteration algorithms to find e-optimal policies are proposed and the proofs of convergence of these two algorithms are given. A numerical example is provided to illustrate the application of the algorithms.
Cascade Markov Decision Processes: Theory and Applications

Manish Gupta

DOI: https://doi.org/10.48550/arXiv.1509.00392

2015-09-02

Abstract:This paper considers the optimal control of time varying continuous time Markov chains whose transition rates are themselves Markov processes. In one set of problems the solution of an ordinary differential equation is shown to determine the optimal performance and feedback controls, while some other cases are shown to lead to singular optimal control problems which are more difficult to solve. Solution techniques are demonstrated using examples from finance to behavioral decision making.

Systems and Control
Sufficiency of Markov Policies for Continuous-Time Jump Markov Decision Processes

Eugene A. Feinberg,Manasa Mandava,Albert N. Shiryaev

DOI: https://doi.org/10.48550/arXiv.2003.01342

2020-05-15

Abstract:This paper extends to Continuous-Time Jump Markov Decision Processes (CTJMDP) the classic result for Markov Decision Processes stating that, for a given initial state distribution, for every policy there is a (randomized) Markov policy, which can be defined in a natural way, such that at each time instance the marginal distributions of state-action pairs for these two policies coincide. It is shown in this paper that this equality takes place for a CTJMDP if the corresponding Markov policy defines a nonexplosive jump Markov process. If this Markov process is explosive, then at each time instance the marginal probability, that a state-action pair belongs to a measurable set of state-action pairs, is not greater for the described Markov policy than the same probability for the original policy. These results are used in this paper to prove that for expected discounted total costs and for average costs per unit time, for a given initial state distribution, for each policy for a CTJMDP the described a Markov policy has the same or better performance.

Optimization and Control
Markov Decision Processes under Risk Sensitivity: A Discount Vanishing Approach

Tanhao Huang,Jinwen Chen

DOI: https://doi.org/10.1016/j.jmaa.2023.128026

IF: 1.417

2024-01-01

Journal of Mathematical Analysis and Applications

Abstract:This paper considers Markov decision processes (MDPs) with risk -sensitivity. The aim is to explore the effects of state transience and non -communication on the optimal control of the system. A vanishing discount approach is investigated. The approximating system is the MDP evaluated by the usual discounted exponential utility. After being appropriately normalized, it is shown that the optimal discounted value functions converge to the optimal risk -sensitive averages as the discount factor goes to 1. These value functions are shown to depend on certain order structure of the state space. In this way it is also proved that the optimal policies for the discounted system converge to the optimal ones for the risk -sensitive system as the discount factor goes to 1. In proving these, an ordered partition of the state space is introduced, which is closely related to the characteristics of state communication, transience and absorption. (c) 2023 Elsevier Inc. All rights reserved.
Mean Field Markov Decision Processes

Nicole Bäuerle

DOI: https://doi.org/10.1007/s00245-023-09985-1

2023-04-12

Abstract:We consider mean-field control problems in discrete time with discounted reward, infinite time horizon and compact state and action space. The existence of optimal policies is shown and the limiting mean-field problem is derived when the number of individuals tends to infinity. Moreover, we consider the average reward problem and show that the optimal policy in this mean-field limit is -optimal for the discounted problem if the number of individuals is large and the discount factor close to one. This result is very helpful, because it turns out that in the special case when the reward does only depend on the distribution of the individuals, we obtain a very interesting subclass of problems where an average reward optimal policy can be obtained by first computing an optimal measure from a static optimization problem and then achieving it with Markov Chain Monte Carlo methods. We give two applications: Avoiding congestion an a graph and optimal positioning on a market place which we solve explicitly.

mathematics, applied
A Discount Vanishing Approximation for Markov Decision Processes with Risk Sensitivity

Tanhao Huang,Xiaoyang Lu,Jinwen Chen

DOI: https://doi.org/10.1007/s10883-024-09691-3

IF: 1.27

2024-04-15

Journal of Dynamical and Control Systems

Abstract:In this paper optimal control of risk-sensitive Markov decision processes with countable states is studied. The state space is not assumed to be communicating. The focus is on dependence of the optimal values on the transition characteristics-communication, transience or absorption. A vanishing discount approach is used to introduce a partition of the state space, and certain transformation of the optimal values under discount is shown to convergence to the optimal values under risk sensitivity, as the discount factor tends to vanish. The partition of the state space turns out to be closely related to the characteristics of state communication, but weights more on the values under discount.

mathematics, applied,automation & control systems
Risk-sensitive discounted Markov decision processes with unbounded reward functions and Borel spaces

Xin Guo

DOI: https://doi.org/10.1080/17442508.2024.2314462

2024-03-14

Stochastics

Abstract:This paper attempts to study the risk-sensitive discounted discrete-time Markov decision processes in Borel spaces, in which the reward functions are allowed to be unbounded from above and from below. We find mild conditions imposed on the primitive data of the decision processes, which not only ensure the existence of a solution to the optimality equation (OE in short), but also are the generalization of the bounded reward case. Furthermore, using the OE and a novel technique, we prove the existence of an optimal policy out of the class of randomized history-dependent policies. Finally, we illustrate our results with an inventory system.

mathematics, applied,statistics & probability
Optimal Time-Abstract Schedulers for CTMDPs and Markov Games

Markus Rabe,Sven Schewe

DOI: https://doi.org/10.4204/EPTCS.28.10

2010-06-26

Abstract:We study time-bounded reachability in continuous-time Markov decision processes for time-abstract scheduler classes. Such reachability problems play a paramount role in dependability analysis and the modelling of manufacturing and queueing systems. Consequently, their analysis has been studied intensively, and techniques for the approximation of optimal control are well understood. From a mathematical point of view, however, the question of approximation is secondary compared to the fundamental question whether or not optimal control exists. We demonstrate the existence of optimal schedulers for the time-abstract scheduler classes for all CTMDPs. Our proof is constructive: We show how to compute optimal time-abstract strategies with finite memory. It turns out that these optimal schedulers have an amazingly simple structure - they converge to an easy-to-compute memoryless scheduling policy after a finite number of steps. Finally, we show that our argument can easily be lifted to Markov games: We show that both players have a likewise simple optimal strategy in these more general structures.

Formal Languages and Automata Theory
Continuous-Time Mean Field Markov Decision Models

Nicole Bäuerle,Sebastian Höfer

DOI: https://doi.org/10.1007/s00245-024-10154-1

2024-06-24

Applied Mathematics & Optimization

Abstract:We consider a finite number of N statistically equal agents, each moving on a finite set of states according to a continuous-time Markov Decision Process (MDP). Transition intensities of the agents and generated rewards depend not only on the state and action of the agent itself, but also on the states of the other agents as well as the chosen action. Interactions like this are typical for a wide range of models in e.g. biology, epidemics, finance, social science and queueing systems among others. The aim is to maximize the expected discounted reward of the system, i.e. the agents have to cooperate as a team. Computationally this is a difficult task when N is large. Thus, we consider the limit for In contrast to other papers we treat this problem from an MDP perspective. This has the advantage that we need less regularity assumptions in order to construct asymptotically optimal strategies than using viscosity solutions of HJB equations. The convergence rate is . We show how to apply our results using two examples: a machine replacement problem and a problem from epidemics. We also show that optimal feedback policies from the limiting problem are not necessarily asymptotically optimal.

mathematics, applied
Markov Decision Processes with Risk-Sensitive Criteria: An Overview

Nicole Bäuerle,Anna Jaśkiewicz

2023-11-13

Abstract:The paper provides an overview of the theory and applications of risk-sensitive Markov decision processes. The term 'risk-sensitive' refers here to the use of the Optimized Certainty Equivalent as a means to measure expectation and risk. This comprises the well-known entropic risk measure and Conditional Value-at-Risk. We restrict our considerations to stationary problems with an infinite time horizon. Conditions are given under which optimal policies exist and solution procedures are explained. We present both the theory when the Optimized Certainty Equivalent is applied recursively as well as the case where it is applied to the cumulated reward. Discounted as well as non-discounted models are reviewed

Risk Management
Markov Decision Processes with Incomplete Information and Semi-Uniform Feller Transition Probabilities

Eugene A. Feinberg,Pavlo O. Kasyanov,Michael Z. Zgurovsky

DOI: https://doi.org/10.48550/arXiv.2108.09232

2022-08-27

Abstract:This paper deals with control of partially observable discrete-time stochastic systems. It introduces and studies Markov Decision Processes with Incomplete Information and with semi-uniform Feller transition probabilities. The important feature of these models is that their classic reduction to Completely Observable Markov Decision Processes with belief states preserves semi-uniform Feller continuity of transition probabilities. Under mild assumptions on cost functions, optimal policies exist, optimality equations hold, and value iterations converge to optimal values for these models. In particular, for Partially Observable Markov Decision Processes the results of this paper imply new and generalize several known sufficient conditions on transition and observation probabilities for weak continuity of transition probabilities for Markov Decision Processes with belief states, the existence of optimal policies, validity of optimality equations defining optimal policies, and convergence of value iterations to optimal values.

Optimization and Control

Continuous Time Markov Decision Processes with Discounted Moment Criterion

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

Mixed Markov Decision Processes in a Semi-Markov Environment with Discounted Criterion

Continuous Time Markov Decision Processes with Nonuniformly Bounded Transition Rate: Expected Total Rewards

Analysis for Some Properties of Discrete Time Markov Decision Processes

Constrained Markov Decision Processes with Non-constant Discount Factor

A survey of recent results on continuous-time Markov decision processes

The Finiteness of the Reward Function and the Optimal Value Function in Markov Decision Processes

Markov Decision Processes with Time-Varying Geometric Discounting

Performance Optimization for Countable Semi-Markov Decision Processes with Discounted-cost

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

Cascade Markov Decision Processes: Theory and Applications

Sufficiency of Markov Policies for Continuous-Time Jump Markov Decision Processes

Markov Decision Processes under Risk Sensitivity: A Discount Vanishing Approach

Mean Field Markov Decision Processes

A Discount Vanishing Approximation for Markov Decision Processes with Risk Sensitivity

Risk-sensitive discounted Markov decision processes with unbounded reward functions and Borel spaces

Optimal Time-Abstract Schedulers for CTMDPs and Markov Games

Continuous-Time Mean Field Markov Decision Models

Markov Decision Processes with Risk-Sensitive Criteria: An Overview

Markov Decision Processes with Incomplete Information and Semi-Uniform Feller Transition Probabilities