A Notation for Markov Decision Processes

Philip S. Thomas,Billy Okal

DOI: https://doi.org/10.48550/arXiv.1512.09075

2016-09-08

Abstract:This paper specifies a notation for Markov decision processes.

Artificial Intelligence

What problem does this paper attempt to address?

Taming denumerable Markov decision processes with decisiveness

Nathalie Bertrand,Patricia Bouyer,Thomas Brihaye,Paulin Fournier

DOI: https://doi.org/10.48550/arXiv.2008.10426

2020-08-24

Abstract:Decisiveness has proven to be an elegant concept for denumerable Markov chains: it is general enough to encompass several natural classes of denumerable Markov chains, and is a sufficient condition for simple qualitative and approximate quantitative model checking algorithms to exist. In this paper, we explore how to extend the notion of decisiveness to Markov decision processes. Compared to Markov chains, the extra non-determinism can be resolved in an adversarial or cooperative way, yielding two natural notions of decisiveness. We then explore whether these notions yield model checking procedures concerning the infimum and supremum probabilities of reachability properties.

Logic in Computer Science
Markov Decision Processes with Ordinal Rewards: Reference Point-Based Preferences

Paul Weng

DOI: https://doi.org/10.1609/icaps.v21i1.13448

2011-03-22

Abstract:In a standard Markov decision process (MDP), rewards are assumed to be precisely known and of quantitative nature. This can be a too strong hypothesis in some situations. When rewards can really be modeled numerically, specifying the reward function is often difficult as it is a cognitively-demanding and/or time-consuming task. Besides, rewards can sometimes be of qualitative nature as when they represent qualitative risk levels for instance. In those cases, it is problematic to use directly standard MDPs and we propose instead to resort to MDPs with ordinal rewards. Only a total order over rewards is assumed to be known. In this setting, we explain how an alternative way to define expressive and interpretable preferences using reference points can be exploited.

Mathematics,Computer Science
Monitored Markov Decision Processes

Simone Parisi,Montaser Mohammedalamen,Alireza Kazemipour,Matthew E. Taylor,Michael Bowling

2024-02-14

Abstract:In reinforcement learning (RL), an agent learns to perform a task by interacting with an environment and receiving feedback (a numerical reward) for its actions. However, the assumption that rewards are always observable is often not applicable in real-world problems. For example, the agent may need to ask a human to supervise its actions or activate a monitoring system to receive feedback. There may even be a period of time before rewards become observable, or a period of time after which rewards are no longer given. In other words, there are cases where the environment generates rewards in response to the agent's actions but the agent cannot observe them. In this paper, we formalize a novel but general RL framework - Monitored MDPs - where the agent cannot always observe rewards. We discuss the theoretical and practical consequences of this setting, show challenges raised even in toy environments, and propose algorithms to begin to tackle this novel setting. This paper introduces a powerful new formalism that encompasses both new and existing problems and lays the foundation for future research.

Machine Learning
J-P: MDP. FP. PP.: Characterizing Total Expected Rewards in Markov Decision Processes as Least Fixed Points with an Application to Operational Semantics of Probabilistic Programs (Technical Report)

Kevin Batz,Benjamin Lucien Kaminski,Christoph Matheja,Tobias Winkler

DOI: https://doi.org/10.1007/978-3-031-75783-9_11

2024-11-26

Abstract:Markov decision processes (MDPs) with rewards are a widespread and well-studied model for systems that make both probabilistic and nondeterministic choices. A fundamental result about MDPs is that their minimal and maximal expected rewards satisfy Bellmann's optimality equations. For various classes of MDPs - notably finite-state MDPs, positive bounded models, and negative models - expected rewards are known to be the least solution of those equations. However, these classes of MDPs are too restrictive for probabilistic program verification. In particular, they assume that all rewards are finite. This is already not the case for the expected runtime of a simple probabilisitic program modeling a 1-dimensional random walk. In this paper, we develop a generalized least fixed point characterization of expected rewards in MDPs without those restrictions. Furthermore, we demonstrate how said characterization can be leveraged to prove weakest-preexpectation-style calculi sound with respect to an operational MDP model.

Logic in Computer Science,Programming Languages
Markov Decision Process Design: A Framework for Integrating Strategic and Operational Decisions

Seth Brown,Saumya Sinha,Andrew J Schaefer

2024-03-22

Abstract:We consider the problem of optimally designing a system for repeated use under uncertainty. We develop a modeling framework that integrates design and operational phases, which are represented by a mixed-integer program and discounted-cost infinite-horizon Markov decision processes, respectively. We seek to simultaneously minimize the design costs and the subsequent expected operational costs. This problem setting arises naturally in several application areas, as we illustrate through examples. We derive a bilevel mixed-integer linear programming formulation for the problem and perform a computational study to demonstrate that realistic instances can be solved numerically.

Optimization and Control,Systems and Control
Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

QY Hu,JY Liu,WY Yue

DOI: https://doi.org/10.1007/3-540-44862-4_8

2003-01-01

Abstract:This paper discusses continuous time Markov decision processes with criterion of expected discounted total rewards, where the state space is countable, the reward rate function is extended real-valued and the discount rate is a real number. Under necessary conditions that the model is well defined, the state space is partitioned into three subsets, on which the optimal value function is positive infinity, negative infinity, or finite, respectively. Correspondingly, the model is reduced into three submodels, by generalizing policies and eliminating some worst actions. Then for the submodel with finite optimal value, the validity of the optimality equation is shown and some its properties are obtained.
Markov Decision Processes with Observation Costs: Framework and Computation with a Penalty Scheme

Christoph Reisinger,Jonathan Tam

DOI: https://doi.org/10.1287/moor.2023.0172

IF: 2.215

2024-05-25

Mathematics of Operations Research

Abstract:We consider Markov decision processes where the state of the chain is only given at chosen observation times and of a cost. Optimal strategies involve the optimization of observation times as well as the subsequent action values. We consider the finite horizon and discounted infinite horizon problems as well as an extension with parameter uncertainty. By including the time elapsed from observations as part of the augmented Markov system, the value function satisfies a system of quasivariational inequalities (QVIs). Such a class of QVIs can be seen as an extension to the interconnected obstacle problem. We prove a comparison principle for this class of QVIs, which implies the uniqueness of solutions to our proposed problem. Penalty methods are then utilized to obtain arbitrarily accurate solutions. Finally, we perform numerical experiments on three applications that illustrate our framework. Funding: J. Tam is supported by the Engineering and Physical Sciences Research Council [Grant 2269738].

mathematics, applied,operations research & management science
Markov Decision Processes with Risk-Sensitive Criteria: An Overview

Nicole Bäuerle,Anna Jaśkiewicz

2023-11-13

Abstract:The paper provides an overview of the theory and applications of risk-sensitive Markov decision processes. The term 'risk-sensitive' refers here to the use of the Optimized Certainty Equivalent as a means to measure expectation and risk. This comprises the well-known entropic risk measure and Conditional Value-at-Risk. We restrict our considerations to stationary problems with an infinite time horizon. Conditions are given under which optimal policies exist and solution procedures are explained. We present both the theory when the Optimized Certainty Equivalent is applied recursively as well as the case where it is applied to the cumulated reward. Discounted as well as non-discounted models are reviewed

Risk Management
Markov Decision Processes with Noisy State Observation

Amirhossein Afsharrad,Sanjay Lall

2023-12-14

Abstract:This paper addresses the challenge of a particular class of noisy state observations in Markov Decision Processes (MDPs), a common issue in various real-world applications. We focus on modeling this uncertainty through a confusion matrix that captures the probabilities of misidentifying the true state. Our primary goal is to estimate the inherent measurement noise, and to this end, we propose two novel algorithmic approaches. The first, the method of second-order repetitive actions, is designed for efficient noise estimation within a finite time window, providing identifiable conditions for system analysis. The second approach comprises a family of Bayesian algorithms, which we thoroughly analyze and compare in terms of performance and limitations. We substantiate our theoretical findings with simulations, demonstrating the effectiveness of our methods in different scenarios, particularly highlighting their behavior in environments with varying stationary distributions. Our work advances the understanding of reinforcement learning in noisy environments, offering robust techniques for more accurate state estimation in MDPs.

Machine Learning,Systems and Control
Optimal decision rules for marked point process models

M.N.M. van Lieshout

2023-09-07

Abstract:We study a Markov decision problem in which the state space is the set of finite marked point configurations in the plane, the actions represent thinnings, the reward is proportional to the mark sum which is discounted over time, and the transitions are governed by a birth-death-growth process. We show that thinning points with large marks is optimal when births follow a Poisson process and marks grow logistically. Explicit values for the thinning threshold and the discounted total expected reward over finite and infinite horizons are also provided. When the points are required to respect a hard core distance, upper and lower bounds on the discounted total expected reward are derived.

Probability
Continuous Time Markov Decision Processes with Nonuniformly Bounded Transition Rate: Expected Total Rewards

Qiying. Hu ∗,jinling. Wang

DOI: https://doi.org/10.1080/02331939808844385

IF: 2.2

1998-01-01

Optimization

Abstract:This paper deals with a stationary continuous time Markov decision process with countable state space, nonuniformly bounded transition rate and extended real valued reward, under the criterion of expected total rewards. The model is transformed into an equivalent discrete time Markov decision process within stochastic stationary policies when the model is well-defined. Thus most results, for example, the optimality equation and the existence of e- optimal policies, in the discrete time Markov decision process may be generalized directly to hold for the continuous time Markov decision process. Furthermore, the state space is partitioned into three subsets: where the optimal value equals to positive infinity, negative infinity, or is finite, respectively
Markov Abstractions for PAC Reinforcement Learning in Non-Markov Decision Processes

Alessandro Ronca,Gabriel Paludo Licks,Giuseppe De Giacomo

DOI: https://doi.org/10.48550/arXiv.2205.01053

2022-05-18

Abstract:Our work aims at developing reinforcement learning algorithms that do not rely on the Markov assumption. We consider the class of Non-Markov Decision Processes where histories can be abstracted into a finite set of states while preserving the dynamics. We call it a Markov abstraction since it induces a Markov Decision Process over a set of states that encode the non-Markov dynamics. This phenomenon underlies the recently introduced Regular Decision Processes (as well as POMDPs where only a finite number of belief states is reachable). In all such kinds of decision process, an agent that uses a Markov abstraction can rely on the Markov property to achieve optimal behaviour. We show that Markov abstractions can be learned during reinforcement learning. Our approach combines automata learning and classic reinforcement learning. For these two tasks, standard algorithms can be employed. We show that our approach has PAC guarantees when the employed algorithms have PAC guarantees, and we also provide an experimental evaluation.

Machine Learning,Artificial Intelligence
Cascade Markov Decision Processes: Theory and Applications

Manish Gupta

DOI: https://doi.org/10.48550/arXiv.1509.00392

2015-09-02

Abstract:This paper considers the optimal control of time varying continuous time Markov chains whose transition rates are themselves Markov processes. In one set of problems the solution of an ordinary differential equation is shown to determine the optimal performance and feedback controls, while some other cases are shown to lead to singular optimal control problems which are more difficult to solve. Solution techniques are demonstrated using examples from finance to behavioral decision making.

Systems and Control
An Overview for Markov Decision Processes in Queues and Networks

Quan-Lin Li,Jing-Yu Ma,Rui-Na Fan,Li Xia

DOI: https://doi.org/10.48550/arXiv.1907.10243

2019-08-23

Abstract:Markov decision processes (MDPs) in queues and networks have been an interesting topic in many practical areas since the 1960s. This paper provides a detailed overview on this topic and tracks the evolution of many basic results. Also, this paper summarizes several interesting directions in the future research. We hope that this overview can shed light to MDPs in queues and networks, and also to their extensive applications in various practical areas.

Optimization and Control,Networking and Internet Architecture
Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet

Marnix Suilen,Thom Badings,Eline M. Bovy,David Parker,Nils Jansen

2024-11-18

Abstract:Markov decision processes (MDPs) are a standard model for sequential decision-making problems and are widely used across many scientific areas, including formal methods and artificial intelligence (AI). MDPs do, however, come with the restrictive assumption that the transition probabilities need to be precisely known. Robust MDPs (RMDPs) overcome this assumption by instead defining the transition probabilities to belong to some uncertainty set. We present a gentle survey on RMDPs, providing a tutorial covering their fundamentals. In particular, we discuss RMDP semantics and how to solve them by extending standard MDP methods such as value iteration and policy iteration. We also discuss how RMDPs relate to other models and how they are used in several contexts, including reinforcement learning and abstraction techniques. We conclude with some challenges for future work on RMDPs.

Artificial Intelligence,Optimization and Control
Reversible Markov decision processes and the Gaussian free field

Venkat Anantharam

DOI: https://doi.org/10.48550/arXiv.2207.05217

2022-07-12

Abstract:A Markov decision problem is called reversible if the stationary controlled Markov chain is reversible under every stationary Markovian strategy. A natural application in which such problems arise is in the control of Metropolis-Hastings type dynamics. We characterize all discrete time reversible Markov decision processes with finite state and actions spaces. We show that policy iteration algorithm for finding an optimal policy can be significantly simplified Markov decision problems of this type. We also highlight the relation between the finite time evolution of the accrual of reward and the Gaussian free field associated to the controlled Markov chain.

Probability,Systems and Control,Optimization and Control
Markov Decision Processes with Incomplete Information and Semi-Uniform Feller Transition Probabilities

Eugene A. Feinberg,Pavlo O. Kasyanov,Michael Z. Zgurovsky

DOI: https://doi.org/10.48550/arXiv.2108.09232

2022-08-27

Abstract:This paper deals with control of partially observable discrete-time stochastic systems. It introduces and studies Markov Decision Processes with Incomplete Information and with semi-uniform Feller transition probabilities. The important feature of these models is that their classic reduction to Completely Observable Markov Decision Processes with belief states preserves semi-uniform Feller continuity of transition probabilities. Under mild assumptions on cost functions, optimal policies exist, optimality equations hold, and value iterations converge to optimal values for these models. In particular, for Partially Observable Markov Decision Processes the results of this paper imply new and generalize several known sufficient conditions on transition and observation probabilities for weak continuity of transition probabilities for Markov Decision Processes with belief states, the existence of optimal policies, validity of optimality equations defining optimal policies, and convergence of value iterations to optimal values.

Optimization and Control
Explicit computations for some Markov modulated counting processes

Michel Mandjes,Peter Spreij

DOI: https://doi.org/10.1007/978-3-319-45875-5

2016-04-10

Abstract:In this paper we present elementary computations for some Markov modulated counting processes, also called counting processes with regime switching. Regime switching has become an increasingly popular concept in many branches of science. In finance, for instance, one could identify the background process with the `state of the economy', to which asset prices react, or as an identification of the varying default rate of an obligor. The key feature of the counting processes in this paper is that their intensity processes are functions of a finite state Markov chain. This kind of processes can be used to model default events of some companies. Many quantities of interest in this paper, like conditional characteristic functions, can all be derived from conditional probabilities, which can, in principle, be analytically computed. We will also study limit results for models with rapid switching, which occur when inflating the intensity matrix of the Markov chain by a factor tending to infinity. The paper is largely expository in nature, with a didactic flavor.

Probability

A Notation for Markov Decision Processes

Taming denumerable Markov decision processes with decisiveness

Markov Decision Processes with Ordinal Rewards: Reference Point-Based Preferences

Monitored Markov Decision Processes

J-P: MDP. FP. PP.: Characterizing Total Expected Rewards in Markov Decision Processes as Least Fixed Points with an Application to Operational Semantics of Probabilistic Programs (Technical Report)

Markov Decision Process Design: A Framework for Integrating Strategic and Operational Decisions

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

Markov Decision Processes with Observation Costs: Framework and Computation with a Penalty Scheme

Markov Decision Processes with Risk-Sensitive Criteria: An Overview

Markov Decision Processes with Noisy State Observation

Optimal decision rules for marked point process models

Continuous Time Markov Decision Processes with Nonuniformly Bounded Transition Rate: Expected Total Rewards

Markov Abstractions for PAC Reinforcement Learning in Non-Markov Decision Processes

Cascade Markov Decision Processes: Theory and Applications

An Overview for Markov Decision Processes in Queues and Networks

Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet

Reversible Markov decision processes and the Gaussian free field

Markov Decision Processes with Incomplete Information and Semi-Uniform Feller Transition Probabilities

Explicit computations for some Markov modulated counting processes