Abstract:This paper studies the average value-at-risk (AVaR) criterion for finite horizon semi-Markov decision processes (SMDPs) in continuous time. Via an alternative representation of AVaR, we reduce the problem of minimizing the AVaR of the finite horizon cost to two subproblems: one is to minimize the expected-positive-deviation of the finite horizon cost from some level over policies, which itself is a new and interesting problem in the finite horizon SMDP setting; the second is an ordinary problem of minimizing a function of a single variable. For the first subproblem, by the technique of extending the state space to include the cost level, we prove that the value function is a minimum solution to the optimality equation, and an optimal policy exists under suitable conditions. Furthermore, we show that the value function is the unique solution in a metric space to the optimality equation when one more condition is imposed, which plays a key role for the algorithm complexity analysis and the policy improvement algorithm. Based on the solution of the first subproblem, the existence and computation of an AVaR optimal policy are established by solving the second subproblem. To facilitate practical implementation of our results, we derive a value iteration algorithm and a policy improvement algorithm for computing an AVaR optimal policy. We perform complexity analysis of the value iteration algorithm, and discuss Monte Carlo simulation as a method of minimizing AVaR for a finite horizon SMDP. To demonstrate our results, two examples about a maintenance system and a cash-flow system are provided.

Mean-Variance Criteria for Finite Continuous-Time Markov Decision Processes

Multivariable Control Performance Assessment Based on Generalized Minimum Variance Benchmark.

Finite Horizon Continuous-Time Markov Decision Processes with Mean and Variance Criteria

Minimum Average Value-at-Risk for Finite Horizon Semi-Markov Decision Processes in Continuous Time.

Markov Decision Processes with Variance Minimization: A New Condition and Approach

Mean-variance optimization of discrete time discounted Markov decision processes.

A Mean–variance Optimization Problem for Discounted Markov Decision Processes

On the First Passage G-Mean-variance Optimality for Discounted Continuous-Time Markov Decision Processes.

Optimization of Markov Decision Processes under the Variance Criterion

Mean-Variance Problems for Finite Horizon Semi-Markov Decision Processes

The Average Variance Criterion for Nonstationary MDP with Borel State Space

Nonstationary Denumerable State Markov Decision Processes – with Average Variance Criterion

Mean-variance optimality for semi-Markov decision processes under first passage criteria.

A Sensitivity‐Based Construction Approach to Variance Minimization of Markov Decision Processes

Optimization Of Parametric Policies Of Markov Decision Processes Under A Variance Criterion

Variance Minimization of Parameterized Markov Decision Processes

Risk‐Sensitive Markov Decision Processes with Combined Metrics of Mean and Variance

Global Algorithms for Mean-Variance Optimization in Markov Decision Processes

A Note on Optimality Conditions for Continuous-Time Markov Decision Processes with Average Cost Criterion

Semi-Markov Decision Processes with Variance Minimization Criterion

Optimality of Mixed Policies for Average Continuous-Time Markov Decision Processes with Constraints