Abstract:Dynamic optimization of mean and variance in Markov decision processes (MDPs) is a long-standing challenge caused by the failure of dynamic programming. In this paper, we propose a new approach to find the globally optimal policy for combined metrics of steady-state mean and variance in an infinite-horizon undiscounted MDP. By introducing the concepts of pseudo mean and pseudo variance, we convert the original problem to a bilevel MDP problem, where the inner one is a standard MDP optimizing pseudo mean-variance and the outer one is a single parameter selection problem optimizing pseudo mean. We use the sensitivity analysis of MDPs to derive the properties of this bilevel problem. By solving inner standard MDPs for pseudo mean-variance optimization, we can identify worse policy spaces dominated by optimal policies of the pseudo problems. We propose an optimization algorithm which can find the globally optimal policy by repeatedly removing worse policy spaces. The convergence and complexity of the algorithm are studied. Another policy dominance property is also proposed to further improve the algorithm efficiency. Numerical experiments demonstrate the performance and efficiency of our algorithms. To the best of our knowledge, our algorithm is the first that efficiently finds the globally optimal policy of mean-variance optimization in MDPs. These results are also valid for solely minimizing the variance metrics in MDPs.

State Entropy Optimization in Markov Decision Processes

Entropy Rate Maximization of Markov Decision Processes under Linear Temporal Logic Tasks

On the Complexity of Computing Maximum Entropy for Markovian Models.

Global Algorithms for Mean-Variance Optimization in Markov Decision Processes

Risk‐Sensitive Markov Decision Processes with Combined Metrics of Mean and Variance

Error bounds of optimization algorithms for semi-Markov decision processes

Optimal Policies for Quantum Markov Decision Processes

Optimal Sample Complexity for Average Reward Markov Decision Processes

Relaxed Equilibria for Time-Inconsistent Markov Decision Processes

The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enough

Optimal models with maximizing probability of first achieving target value in the preceding stages

Risk probability optimization of finite horizon piecewise deterministic Markov decision processes

Entropy Maximization for Partially Observable Markov Decision Processes

Finding Optimal Memoryless Policies of POMDPs under the Expected Average Reward Criterion

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

How to Explore with Belief: State Entropy Maximization in POMDPs

Entropy-Regularized Stochastic Games

Analysis for Some Properties of Discrete Time Markov Decision Processes

Predictable Interval MDPs through Entropy Regularization

Landscape of Policy Optimization for Finite Horizon MDPs with General State and Action

Partial and Conditional Expectations in Markov Decision Processes with Integer Weights