Abstract:CVaR (Conditional value at risk) is a risk metric widely used in finance. However, dynamically optimizing CVaR is difficult, because it is not a standard Markov decision process (MDP) and the principle of dynamic programming fails. In this paper, we study the infinite‐horizon discrete‐time MDP with a long‐run CVaR criterion, from the view of sensitivity‐based optimization. By introducing a pseudo CVaR metric, we reformulate the problem as a bilevel MDP model and derive a CVaR difference formula which quantifies the difference of long‐run CVaR under any two policies. The optimality of deterministic policies is derived. We obtain a so‐called Bellman local optimality equation for CVaR, which is a necessary and sufficient condition for locally optimal policies and only necessary for globally optimal policies. A CVaR derivative formula is also derived for providing more sensitivity information. Then we develop a policy iteration type algorithm to efficiently optimize CVaR, which is shown to converge to a local optimum in mixed policy space. Furthermore, based on the sensitivity analysis of our bilevel MDP formulation and critical points, we develop a globally optimal algorithm. The piecewise linearity and segment convexity of the optimal pseudo CVaR function are also established. Our main results and algorithms are further extended to optimize the mean and CVaR simultaneously. Finally, we conduct numerical experiments relating to portfolio management to demonstrate the main results. Our work sheds light on dynamically optimizing CVaR from a sensitivity viewpoint. This article is protected by copyright. All rights reserved

Risk-averse learning with delayed feedback

Risk-averse Learning with Non-Stationary Distributions

Risk-Averse No-Regret Learning in Online Convex Games

Learning of Nash Equilibria in Risk-Averse Games

Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaR

Risk-Sensitive Reinforcement Learning: Iterated CVaR and the Worst Path.

Dynamic Regret with Unknown Delays

Risk-Sensitive Online Algorithms

Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Path

Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback

Risk‐sensitive markov decision processes with long‐run CVaR criterion

Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach

Worse-Case Conditional Value-at-Risk for Asymmetrically Distributed Asset Scenarios Returns

Posterior Sampling with Delayed Feedback for Reinforcement Learning with Linear Function Approximation

Earning and Learning with Varying Cost

Risk-Averse Reinforcement Learning via Dynamic Time-Consistent Risk Measures

Provably Efficient CVaR RL in Low-rank MDPs

Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Risk

Distributionally Time-Varying Online Stochastic Optimization under Polyak-Łojasiewicz Condition with Application in Conditional Value-at-Risk Statistical Learning

Optimal Control of Conditional Value-at-Risk in Continuous Time

√N-Regret for Learning in Markov Decision Processes with Function Approximation and Low Bellman Rank.