Abstract:CVaR (Conditional value at risk) is a risk metric widely used in finance. However, dynamically optimizing CVaR is difficult, because it is not a standard Markov decision process (MDP) and the principle of dynamic programming fails. In this paper, we study the infinite‐horizon discrete‐time MDP with a long‐run CVaR criterion, from the view of sensitivity‐based optimization. By introducing a pseudo CVaR metric, we reformulate the problem as a bilevel MDP model and derive a CVaR difference formula which quantifies the difference of long‐run CVaR under any two policies. The optimality of deterministic policies is derived. We obtain a so‐called Bellman local optimality equation for CVaR, which is a necessary and sufficient condition for locally optimal policies and only necessary for globally optimal policies. A CVaR derivative formula is also derived for providing more sensitivity information. Then we develop a policy iteration type algorithm to efficiently optimize CVaR, which is shown to converge to a local optimum in mixed policy space. Furthermore, based on the sensitivity analysis of our bilevel MDP formulation and critical points, we develop a globally optimal algorithm. The piecewise linearity and segment convexity of the optimal pseudo CVaR function are also established. Our main results and algorithms are further extended to optimize the mean and CVaR simultaneously. Finally, we conduct numerical experiments relating to portfolio management to demonstrate the main results. Our work sheds light on dynamically optimizing CVaR from a sensitivity viewpoint. This article is protected by copyright. All rights reserved

A Sensitivity‐Based Construction Approach to Variance Minimization of Markov Decision Processes

A sensitivity-based construction approach to sample-path variance minimization of Markov decision processes

Variance Minimization of Parameterized Markov Decision Processes

Multivariable Control Performance Assessment Based on Generalized Minimum Variance Benchmark.

Optimization of Markov Decision Processes under the Variance Criterion

Optimization Of Parametric Policies Of Markov Decision Processes Under A Variance Criterion

Markov Decision Processes with Variance Minimization: A New Condition and Approach

Risk‐Sensitive Markov Decision Processes with Combined Metrics of Mean and Variance

An Iterative Approach to Reduce the Variance of Stochastic Dynamic Systems

Mean-variance optimization of discrete time discounted Markov decision processes.

A New Method for Mean-Variance Optimization of Stochastic Dynamic Systems.

Semi-Markov Decision Processes with Variance Minimization Criterion

Mean-Variance Criteria for Finite Continuous-Time Markov Decision Processes

Markov Decision Processes under Risk Sensitivity: A Discount Vanishing Approach

A unified approach for semi-Markov decision processes with discounted and average reward criteria

Mean-variance optimality for semi-Markov decision processes under first passage criteria.

Risk‐sensitive markov decision processes with long‐run CVaR criterion

Finite Horizon Continuous-Time Markov Decision Processes with Mean and Variance Criteria

Global Algorithms for Mean-Variance Optimization in Markov Decision Processes

A Mean–variance Optimization Problem for Discounted Markov Decision Processes

A Discount Vanishing Approximation for Markov Decision Processes with Risk Sensitivity