Abstract:Markov Decision Processes (MDPs) are mathematical models of sequential decision-making under uncertainty that have found applications in healthcare, manufacturing, logistics, and others. In these models, a decision-maker observes the state of a stochastic process and determines which action to take with the goal of maximizing the expected total discounted rewards received. In many applications, the state space of the true system is large and there may be limited observations out of certain states to estimate the transition probability matrix. To overcome this, modelers will aggregate the true states into ``superstates" resulting in a smaller state space. This aggregation process improves computational tractability and increases the number of observations among superstates. Thus, the modeler's choice of state space leads to a trade-off in transition probability estimates. While coarser discretization of the state space gives more observations in each state to estimate the transition probability matrix, this comes at the cost of precision in the state characterization and resulting policy recommendations. In this paper, we consider the implications of this modeling decision on the resulting policies from MDPs for which the true model is expected to have a threshold policy that is optimal. We analyze these MDPs and provide conditions under which the aggregated MDP will also have an optimal threshold policy. Using a simulation study, we explore the trade-offs between more fine and more coarse aggregation. We explore the the show that there is the highest potential for policy improvement on larger state spaces, but that aggregated MDPs are preferable under limited data. We discuss how these findings the implications of our findings for modelers who must select which state space design to use.

State Aggregation In Markov Decision Processes

The implications of state aggregation in deteriorating Markov Decision Processes with optimal threshold policies

A Unified Approach to Time-Aggregated Markov Decision Processes.

Average cost Markov decision processes with countable state spaces

Markov decision Processes with fractional costs

On State Aggregation to Approximate Complex Value Functions in Large-Scale Markov Decision Processes.

Performance Loss Bound for State Aggregation in a Class of Supply Demand Matching Systems

Risk-sensitive Average Continuous-Time Markov Decision Processes with Unbounded Transition and Cost Rates.

Markov Decision Problems with Unbounded Transition Rates under Discounted-Cost Performance Criteria

A Lazy Abstraction Algorithm for Markov Decision Processes: Theory and Initial Evaluation

Convergence of Markov Decision Processes with Constraints and State-Action Dependent Discount Factors

A Note on the Existence of Optimal Stationary Policies for Average Markov Decision Processes with Countable States

Average Continuous Control of Piecewise Deterministic Markov Processes

Solution to the risk-sensitive average cost optimality equation in a class of Markov decision processes with finite state space

Adaptive Low-Nonnegative-Rank Approximation for State Aggregation of Markov Chains

Weighted mesh algorithms for general Markov decision processes: Convergence and tractability

State Entropy Optimization in Markov Decision Processes

Bayesian Learning of Optimal Policies in Markov Decision Processes with Countably Infinite State-Space

Nonstationary Denumerable State Markov Decision Processes – with Average Variance Criterion

Denumerable Continuous-Time Markov Decision Processes with Multiconstraints on Average Costs

Discretized Approximations for POMDP with Average Cost