Abstract:This paper studies the minimizing risk problems in Markov decision processes with countable state space and reward set. The objective is to find a policy which minimizes the probability (risk) that the total discounted rewards do not exceed a specified value (target). In this sort of model, the decision made by the decision maker depends not only on system's states, but also on his target values. By introducing the decision-maker's state, we formulate a framework for minimizing risk models. The policies discussed depend on target values and the rewards may be arbitrary real numbers. For the finite horizon model, the main results obtained are: (i) The optimal value functions are distribution functions of the target, (ii) there exists an optimal deterministic Markov policy, and (iii) a policy is optimal if and only if at each realizable state it always takes optimal action. In addition, we obtain a sufficient condition and a necessary condition for the existence of finite horizon optimal policy independent of targets and we give an algorithm computing finite horizon optimal policies and optimal value functions. For an infinite horizon model, we establish the optimality equation and we obtain the structure property of optimal policy. We prove that the optimal value function is a distribution function of target and we present a new approximation formula which is the generalization of the nonnegative rewards cases. An example which illustrates the mistakes of previous literature shows that the existence of optimal policy has not been proved really. In this paper, we give an existence condition, which is a sufficient and necessary condition for the existence of an infinite horizon optimal policy independent of targets, and we point out that whether there exists an optimal policy remains an open problem in the general case.

Optimal decision rules for marked point process models

Markov Decision Processes with State-Dependent Discount Factors and Unbounded Rewards/costs.

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

The Finiteness of the Reward Function and the Optimal Value Function in Markov Decision Processes

Denumerable-state Continuous-Time Markov Decision Processes with Unbounded Transition and Reward Rates under the Discounted Criterion

Continuous Time Markov Decision Processes with Nonuniformly Bounded Transition Rate: Expected Total Rewards

Minimizing Risk Models in Markov Decision Processes with Policies Depending on Target Values

Nonstationary Markov Decision Processes with Risk Probability Criteria

Mean Field Markov Decision Processes

Risk-sensitive Continuous-Time Markov Decision Processes with Unbounded Rates and Borel Spaces

Continuous-Time Markov Decision Processes with Unbounded Transition and Discounted-Reward Rates

Risk-sensitive discounted Markov decision processes with unbounded reward functions and Borel spaces

Markov Decision Problems with Unbounded Transition Rates under Discounted-Cost Performance Criteria

Analysis for Some Properties of Discrete Time Markov Decision Processes

Nonhomogeneous Markov Decision Processes with Borel State Space-The Average Criterion with Nonuniformly Bounded Rewards.

Maximizing the probability of visiting a set infinitely often for a Markov decision process with Borel state and action spaces

Finite Horizon Partially Observable Semi-Markov Decision Processes under Risk Probability Criteria

Minimizing Risk Models in Denumerable Semi-Markov Decision Processes with A Target Set

Extremal behavior of stationary marked point processes

Optimal models with maximizing probability of first achieving target value in the preceding stages

Continuous-Time Markov Decision Processes with State-Dependent Discount Factors