Abstract:This paper studies the risk-averse mean-variance optimization in infinite-horizon discounted Markov decision processes (MDPs). The involved variance metric concerns reward variability during the whole process, and future deviations are discounted to their present values. This discounted mean-variance optimization yields a reward function dependent on a discounted mean, and this dependency renders traditional dynamic programming methods inapplicable since it suppresses a crucial property -- time consistency. To deal with this unorthodox problem, we introduce a pseudo mean to transform the untreatable MDP to a standard one with a redefined reward function in standard form and derive a discounted mean-variance performance difference formula. With the pseudo mean, we propose a unified algorithm framework with a bilevel optimization structure for the discounted mean-variance optimization. The framework unifies a variety of algorithms for several variance-related problems including, but not limited to, risk-averse variance and mean-variance optimizations in discounted and average MDPs. Furthermore, the convergence analyses missing from the literature can be complemented with the proposed framework as well. Taking the value iteration as an example, we develop a discounted mean-variance value iteration algorithm and prove its convergence to a local optimum with the aid of a Bellman local-optimality equation. Finally, we conduct a numerical experiment on portfolio management to validate the proposed algorithm.

On the First Passage G-Mean-variance Optimality for Discounted Continuous-Time Markov Decision Processes.

First Passage Markov Decision Processes with Constraints and Varying Discount Factors

A Mean–variance Optimization Problem for Discounted Markov Decision Processes

First Passage Optimality for Continuous-Time Markov Decision Processes with Varying Discount Factors and History-Dependent Policies

Mean-variance optimality for semi-Markov decision processes under first passage criteria.

First Passage Models for Denumerable Semi-Markov Decision Processes with Nonnegative Discounted Costs

Mean-variance optimization of discrete time discounted Markov decision processes.

Mean-Variance Criteria for Finite Continuous-Time Markov Decision Processes

Optimization of Markov Decision Processes under the Variance Criterion

Mean-Variance Problems for Finite Horizon Semi-Markov Decision Processes

Nonstationary Denumerable State Markov Decision Processes – with Average Variance Criterion

Constrained Markov Decision Processes with First Passage Criteria

A Unified Algorithm Framework for Mean-Variance Optimization in Discounted Markov Decision Processes

Finite Horizon Continuous-Time Markov Decision Processes with Mean and Variance Criteria

Global Algorithms for Mean-Variance Optimization in Markov Decision Processes

Semi-Markov Decision Processes with Variance Minimization Criterion

The Risk Probability Criterion for Discounted Continuous-Time Markov Decision Processes

New Discount and Average Optimality Conditions for Continuous-Time Markov Decision Processes

Markov Decision Processes with Variance Minimization: A New Condition and Approach

Finite-horizon Optimality for Continuous-Time Markov Decision Processes with Unbounded Transition Rates

First Passage Risk Probability Minimization for Piecewise Deterministic Markov Decision Processes