Abstract:This paper studies the average value-at-risk (AVaR) criterion for finite horizon semi-Markov decision processes (SMDPs) in continuous time. Via an alternative representation of AVaR, we reduce the problem of minimizing the AVaR of the finite horizon cost to two subproblems: one is to minimize the expected-positive-deviation of the finite horizon cost from some level over policies, which itself is a new and interesting problem in the finite horizon SMDP setting; the second is an ordinary problem of minimizing a function of a single variable. For the first subproblem, by the technique of extending the state space to include the cost level, we prove that the value function is a minimum solution to the optimality equation, and an optimal policy exists under suitable conditions. Furthermore, we show that the value function is the unique solution in a metric space to the optimality equation when one more condition is imposed, which plays a key role for the algorithm complexity analysis and the policy improvement algorithm. Based on the solution of the first subproblem, the existence and computation of an AVaR optimal policy are established by solving the second subproblem. To facilitate practical implementation of our results, we derive a value iteration algorithm and a policy improvement algorithm for computing an AVaR optimal policy. We perform complexity analysis of the value iteration algorithm, and discuss Monte Carlo simulation as a method of minimizing AVaR for a finite horizon SMDP. To demonstrate our results, two examples about a maintenance system and a cash-flow system are provided.

Finite Horizon Continuous-Time Markov Decision Processes with Mean and Variance Criteria

Mean-Variance Problems for Finite Horizon Semi-Markov Decision Processes

Mean-variance optimality for semi-Markov decision processes under first passage criteria.

Mean-variance optimization of discrete time discounted Markov decision processes.

A Mean–variance Optimization Problem for Discounted Markov Decision Processes

Risk Probability Minimization Problems for Continuous-Time Markov Decision Processes on Finite Horizon

Mean-Variance Criteria for Finite Continuous-Time Markov Decision Processes

Semi-Markov Decision Processes with Variance Minimization Criterion

Finite-horizon Optimality for Continuous-Time Markov Decision Processes with Unbounded Transition Rates

Minimum Average Value-at-Risk for Finite Horizon Semi-Markov Decision Processes in Continuous Time.

An Optimistic Value Iteration for Mean–variance Optimization in Discounted Markov Decision Processes

Optimization of Markov Decision Processes under the Variance Criterion

Finite Horizon Partially Observable Semi-Markov Decision Processes under Risk Probability Criteria

Minimum Risk Probability for Finite Horizon Semi-Markov Decision Processes

Markov Decision Processes with Variance Minimization: A New Condition and Approach

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

The Risk Probability Criterion for Discounted Continuous-Time Markov Decision Processes

Variance Minimization of Parameterized Markov Decision Processes

Mean Field Markov Decision Processes

Risk-sensitive Continuous-Time Markov Decision Processes with Unbounded Rates and Borel Spaces

Risk‐Sensitive Markov Decision Processes with Combined Metrics of Mean and Variance