Abstract:A large class of decision making under uncertainty problems can be described via Markov decision processes (MDPs) or partially observable MDPs (POMDPs), with application to artificial intelligence and operations research, among others. Traditionally, policy synthesis techniques are proposed such that a total expected cost or reward is minimized or maximized. However, optimality in the total expected cost sense is only reasonable if system behavior in the large number of runs is of interest, which has limited the use of such policies in practical mission-critical scenarios, wherein large deviations from the expected behavior may lead to mission failure. In this paper, we consider the problem of designing policies for MDPs and POMDPs with objectives and constraints in terms of dynamic coherent risk measures, which we refer to as the constrained risk-averse problem. For MDPs, we reformulate the problem into a infsup problem via the Lagrangian framework and propose an optimization-based method to synthesize Markovian policies. For MDPs, we demonstrate that the formulated optimization problems are in the form of difference convex programs (DCPs) and can be solved by the disciplined convex-concave programming (DCCP) framework. We show that these results generalize linear programs for constrained MDPs with total discounted expected costs and constraints. For POMDPs, we show that, if the coherent risk measures can be defined as a Markov risk transition mapping, an infinite-dimensional optimization can be used to design Markovian belief-based policies. For stochastic finite-state controllers (FSCs), we show that the latter optimization simplifies to a (finite-dimensional) DCP and can be solved by the DCCP framework. We incorporate these DCPs in a policy iteration algorithm to design risk-averse FSCs for POMDPs.

Dynamic Optimization with Timing Risk

Risk-Adaptive Approaches to Stochastic Optimization: A Survey

Optimal execution with dynamic risk adjustment

Dynamic portfolio optimization with liquidity cost and market impact: a simulation-and-regression approach

Market timing, investment, and risk management

On Risk-Sensitive Decision Making Under Uncertainty

Constrained Reinforcement Learning for Dynamic Optimization under Uncertainty

Dynamic Index Tracking and Risk Exposure Control Using Derivatives

A stochastic programming model for money management

A data-driven approach for a class of stochastic dynamic optimization problems

The Value of Timing Risk

An overview of optimal control optimization problems driven by non-convexity measures

Risk-averse risk-constrained optimal control

Risk conscious investment¶

Dynamic margin optimization

Dynamic optimization of a portfolio

Stochastic Control for Optimal Execution: Fast Approximation Solution Scheme under Nested Mean-semi Deviation and Conditional Value at Risk

Stochastic control of optimized certainty equivalents

A Course in Dynamic Optimization

Risk-Averse Decision Making Under Uncertainty

Optimizing Execution Cost Using Stochastic Control