Abstract:CVaR (Conditional value at risk) is a risk metric widely used in finance. However, dynamically optimizing CVaR is difficult, because it is not a standard Markov decision process (MDP) and the principle of dynamic programming fails. In this paper, we study the infinite‐horizon discrete‐time MDP with a long‐run CVaR criterion, from the view of sensitivity‐based optimization. By introducing a pseudo CVaR metric, we reformulate the problem as a bilevel MDP model and derive a CVaR difference formula which quantifies the difference of long‐run CVaR under any two policies. The optimality of deterministic policies is derived. We obtain a so‐called Bellman local optimality equation for CVaR, which is a necessary and sufficient condition for locally optimal policies and only necessary for globally optimal policies. A CVaR derivative formula is also derived for providing more sensitivity information. Then we develop a policy iteration type algorithm to efficiently optimize CVaR, which is shown to converge to a local optimum in mixed policy space. Furthermore, based on the sensitivity analysis of our bilevel MDP formulation and critical points, we develop a globally optimal algorithm. The piecewise linearity and segment convexity of the optimal pseudo CVaR function are also established. Our main results and algorithms are further extended to optimize the mean and CVaR simultaneously. Finally, we conduct numerical experiments relating to portfolio management to demonstrate the main results. Our work sheds light on dynamically optimizing CVaR from a sensitivity viewpoint. This article is protected by copyright. All rights reserved

Controlled Markov Chains with AVaR Criteria for Unbounded Costs

Minimum Average Value-at-Risk for Finite Horizon Semi-Markov Decision Processes in Continuous Time.

Zero-sum Stochastic Games with the Average-Value-at-risk Criterion

Controlled Semi-Markov Chains with Risk-Sensitive Average Cost Criterion

Characterization of the optimal average cost in Markov decision chains driven by a risk-seeking controller

Nonstationary Denumerable State Markov Decision Processes – with Average Variance Criterion

Risk-Averse Control of Undiscounted Transient Markov Models

Constrained Continuous-Time Markov Decision Processes with Average Criteria

Risk-sensitive Markov control processes

Risk-sensitive Average Continuous-Time Markov Decision Processes with Unbounded Transition and Cost Rates.

Solution to the risk-sensitive average cost optimality equation in a class of Markov decision processes with finite state space

Average cost Markov decision processes with countable state spaces

Average cost optimal control under weak ergodicity hypotheses: Relative value iterations

Optimal Control of Ergodic Continuous-Time Markov Chains with Average Sample-Path Rewards

On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs

Risk-Sensitive Optimal Control in Communicating Average Markov Decision Chains

Limiting Average Criteria for Nonstationary Markov Decision Processes

Mean-Variance Criteria for Finite Continuous-Time Markov Decision Processes

On Average Risk-sensitive Markov Control Processes

Constrained Risk-Averse Markov Decision Processes

Risk‐sensitive markov decision processes with long‐run CVaR criterion