Abstract:This paper studies the minimizing risk problems in Markov decision processes with countable state space and reward set. The objective is to find a policy which minimizes the probability (risk) that the total discounted rewards do not exceed a specified value (target). In this sort of model, the decision made by the decision maker depends not only on system's states, but also on his target values. By introducing the decision-maker's state, we formulate a framework for minimizing risk models. The policies discussed depend on target values and the rewards may be arbitrary real numbers. For the finite horizon model, the main results obtained are: (i) The optimal value functions are distribution functions of the target, (ii) there exists an optimal deterministic Markov policy, and (iii) a policy is optimal if and only if at each realizable state it always takes optimal action. In addition, we obtain a sufficient condition and a necessary condition for the existence of finite horizon optimal policy independent of targets and we give an algorithm computing finite horizon optimal policies and optimal value functions. For an infinite horizon model, we establish the optimality equation and we obtain the structure property of optimal policy. We prove that the optimal value function is a distribution function of target and we present a new approximation formula which is the generalization of the nonnegative rewards cases. An example which illustrates the mistakes of previous literature shows that the existence of optimal policy has not been proved really. In this paper, we give an existence condition, which is a sufficient and necessary condition for the existence of an infinite horizon optimal policy independent of targets, and we point out that whether there exists an optimal policy remains an open problem in the general case.

On Maximizing Probabilities for Over-Performing a Target for Markov Decision Processes

Duality Between Large Deviation Control and Risk-Sensitive Control for Markov Decision Processes.

Dual Adaptive Control

Minimizing Risk Models in Markov Decision Processes with Policies Depending on Target Values

Risk-Sensitivity Vanishing Limit for Controlled Markov Processes

Risk Aware Minimum Principle for Optimal Control of Stochastic Differential Equations

Risk‐sensitive maximum principle for stochastic optimal control of mean‐field type Markov regime‐switching jump‐diffusion systems

Optimal models with maximizing probability of first achieving target value in the preceding stages

Probabilistic Control and Majorization of Optimal Control

On the Maximization of Long-Run Reward CVaR for Markov Decision Processes

Markov Decision Processes under Risk Sensitivity: A Discount Vanishing Approach

A Discount Vanishing Approximation for Markov Decision Processes with Risk Sensitivity

On Average Risk-sensitive Markov Control Processes

Sufficient Stochastic Maximum Principle for the Optimal Control of Semi-Markov Modulated Jump-Diffusion with Application to Financial Optimization

Integrating Risk-Averse and Constrained Reinforcement Learning for Robust Decision-Making in High-StakesScenarios

Risk probability optimization of finite horizon piecewise deterministic Markov decision processes

Optimal control of probabilistic discrete event systems on Markov decision processes

Risk-Averse Decision Making Under Uncertainty

Maximum Principle for the Stochastic Optimal Control Problem with Delay and Application

Primal-Dual Regression Approach for Markov Decision Processes with General State and Action Spaces

Reachability and Safety Objectives in Markov Decision Processes on Long but Finite Horizons