Abstract:We build on the theory of capital requirements (CRs) to create a new framework for modeling dynamic risk preferences. The key question is how to evaluate the risk of a payoff stream sequentially as new information is revealed. In our model, we associate each payoff stream with a disbursement strategy and a premium schedule to form a triple of stochastic processes. We characterize risk preferences in terms of a single set that we call the risk frontier which characterizes acceptable triples. We then propose the generalized capital requirement (GCR) which evaluates the risk of a payoff stream by minimizing the premium schedule over acceptable triples. We apply this model to a risk-aware decision maker (DM) who controls a Markov decision process (MDP) and wants to find a policy to minimize the GCR of its payoff stream. The resulting GCR-MDP recovers many well-known risk-aware MDPs as special cases. To make this approach computationally viable, we obtain the temporal decomposition of the GCR in terms of the risk frontier. Then, we connect the temporal decomposition with the notion of an information state to compactly capture the dependence of DM's risk preferences on the problem history, where augmented dynamic programming can be used to compute an optimal policy. We report numerical experiments for the GCR-minimizing newsvendor.

What problem does this paper attempt to address?

The paper aims to address the issue of risk management in dynamic optimization problems, specifically how to effectively assess and handle risks in Markov Decision Processes (MDP). The core of the research lies in proposing a new framework—Generalized Capital Requirements (GCR)—to model dynamic risk preferences and apply it to risk-sensitive decision-making processes. The main contributions include: 1. **New Family of Generalized Capital Requirements (GCR)**: Based on the theory of capital requirements, the paper introduces the concept of Generalized Capital Requirements by integrating three key components (acceptance set, financial strategy set, cost function) into a single risk frontier set. This allows risk assessment to be completed by minimizing the premium in the acceptable triplet (revenue stream, payment strategy, premium plan). 2. **Time Consistency**: The authors identify sufficient conditions for the time consistency of GCR and use lattice theory to prove the time decomposition of GCR, which means GCR can be obtained in a recursive form. 3. **Risk-Sensitive Information State**: The paper introduces the concept of information state to compactly represent the decision-maker's risk preferences and their changes over time. This concept is crucial for handling risk preferences that depend on historical information. 4. **New Family of GCR-MDP**: The paper creates a new class of risk-sensitive MDPs, called GCR-MDP, where the objective is to find a policy to minimize the GCR of the revenue stream. Additionally, a dynamic programming decomposition of GCR-MDP is provided, which helps in computing the optimal policy. 5. **Unified Framework**: The paper demonstrates that several known risk-sensitive MDPs can be recast as GCR-MDPs, such as nested risk measures, expected utility maximization, conditional value at risk, etc. This unified framework highlights the common structure among these models and simplifies the design of risk-sensitive MDPs. 6. **Extensions and Applications**: The paper also discusses new GCR-MDP models based on wealth-dependent preferences and target shortfall, and validates the effectiveness of the approach through numerical experiments. In summary, the main goal of this paper is to extend existing risk management theories by proposing the GCR framework and applying it to risk-sensitive Markov Decision Processes, providing an effective method for handling risk management issues in dynamic environments.

Dynamic Capital Requirements for Markov Decision Processes

Acceptable risks and related decision problems with multiple risk-averse agents

Risk-sensitive Markov Decision Process and Learning under General Utility Functions

Constrained Risk-Averse Markov Decision Processes

Risk-Averse Markov Decision Processes Through a Distributional Lens

Risk Sensitive Markov Decision Process for Portfolio Management

Markov Decision Processes with Risk-Sensitive Criteria: An Overview

Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes

Deterministic definition of the capital risk

Risk-Averse Decision Making Under Uncertainty

Risk-Averse Approximate Dynamic Programming with Quantile-Based Risk Measures

Dynamic capital allocation rules via BSDEs: an axiomatic approach

Risk-Sensitive ICAPM With Application to Fixed-Income Management

Risk-Averse Reinforcement Learning via Dynamic Time-Consistent Risk Measures

Risk‐sensitive markov decision processes with long‐run CVaR criterion

Markov Decision Processes with Time-Varying Geometric Discounting

Model and Reinforcement Learning for Markov Games with Risk Preferences

Dynamic Reinsurance in Discrete Time Minimizing the Insurer's Cost of Capital

Robustness to Modeling Errors in Risk-Sensitive Markov Decision Problems with Markov Risk Measures