Abstract:A large class of decision making under uncertainty problems can be described via Markov decision processes (MDPs) or partially observable MDPs (POMDPs), with application to artificial intelligence and operations research, among others. Traditionally, policy synthesis techniques are proposed such that a total expected cost or reward is minimized or maximized. However, optimality in the total expected cost sense is only reasonable if system behavior in the large number of runs is of interest, which has limited the use of such policies in practical mission-critical scenarios, wherein large deviations from the expected behavior may lead to mission failure. In this paper, we consider the problem of designing policies for MDPs and POMDPs with objectives and constraints in terms of dynamic coherent risk measures, which we refer to as the constrained risk-averse problem. For MDPs, we reformulate the problem into a infsup problem via the Lagrangian framework and propose an optimization-based method to synthesize Markovian policies. For MDPs, we demonstrate that the formulated optimization problems are in the form of difference convex programs (DCPs) and can be solved by the disciplined convex-concave programming (DCCP) framework. We show that these results generalize linear programs for constrained MDPs with total discounted expected costs and constraints. For POMDPs, we show that, if the coherent risk measures can be defined as a Markov risk transition mapping, an infinite-dimensional optimization can be used to design Markovian belief-based policies. For stochastic finite-state controllers (FSCs), we show that the latter optimization simplifies to a (finite-dimensional) DCP and can be solved by the DCCP framework. We incorporate these DCPs in a policy iteration algorithm to design risk-averse FSCs for POMDPs.

Markov Decision Processes Formulation for Stochastic and Dynamic Bank Branches Location Problems

Online Markov decision processes with policy iteration

Markov Decision Process Design: A Framework for Integrating Strategic and Operational Decisions

A Discrete Event Simulation Based Production Line Optimization through Markov Decision Process.

Approximate Policy Iteration for Robust Stochastic Control of Multi-agent Markov Decision Processes

Dynamic portfolio optimization under multi-factor model in stochastic markets

A Branch-and-Cut Method for Dynamic Decision Making Under Joint Chance Constraints

A robust optimization approach to multi-period competitive location problem for bank branches considering first-mover advantage

Constrained Risk-Averse Markov Decision Processes

Bank Queuing Optimization Based on Markov Process

A dynamic model of functioning of a bank

Risk-Averse Decision Making Under Uncertainty

A stochastic programming model for money management

A risk-aware maintenance model based on a constrained Markov decision process

Modeling and Optimizing Resource Allocation Decisions through Multi-model Markov Decision Processes with Capacity Constraints

Stochastic Optimization Based On Principal-Agent Problem

A dynamic stochastic programming model for international portfolio management

A Potential-Based Method for Finite-Stage Markov Decision Process

Markov Decision Process and Approximate Dynamic Programming for a Patient Assignment Scheduling problem

Unified Framework of Mean-Field Formulations for Optimal Multi-Period Mean-Variance Portfolio Selection

Risk Sensitive Markov Decision Process for Portfolio Management