Abstract:This paper investigates the optimization problem of an infinite stage discrete time Markov decision process (MDP) with a long‐run average metric considering both mean and variance of rewards together. Such performance metric is important since the mean indicates average returns and the variance indicates risk or fairness. However, the variance metric couples the rewards at all stages, the traditional dynamic programming is inapplicable as the principle of time consistency fails. We study this problem from a new perspective called the sensitivity‐based optimization theory. A performance difference formula is derived and it can quantify the difference of the mean‐variance combined metrics of MDPs under any two different policies. The difference formula can be utilized to generate new policies with strictly improved mean‐variance performance. A necessary condition of the optimal policy and the optimality of deterministic policies are derived. We further develop an iterative algorithm with a form of policy iteration, which is proved to converge to local optima both in the mixed and randomized policy space. Specially, when the mean reward is constant in policies, the algorithm is guaranteed to converge to the global optimum. Finally, we apply our approach to study the fluctuation reduction of wind power in an energy storage system, which demonstrates the potential applicability of our optimization method.

A unified approach for semi-Markov decision processes with discounted and average reward criteria

A Unified Approach to Markov Decision Problems and Performance Sensitivity Analysis with Discounted and Average Criteria: Multichain Cases

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

A Survey on Semi-Markov Decision Processes

Mixed Markov Decision Processes in a Semi-Markov Environment with Discounted Criterion

Performance Optimization for Countable Semi-Markov Decision Processes with Discounted-cost

Reinforcement learning algorithms for semi-Markov decision processes with average reward

Error bounds of optimization algorithms for semi-Markov decision processes

Optimization Algorithms for Semi-Markov Control Processes with Average Criteria

A Unified Approach to Time-Aggregated Markov Decision Processes.

RVI reinforcement learning for semi-Markov decision processes with average reward

Relations Between Discounted Models and Average Models for Semi-Markov Decision Processes

Risk‐Sensitive Markov Decision Processes with Combined Metrics of Mean and Variance

Optimal Stationary Policies for Semi-Markov Control Processes with Discounted-Cost Criteria

An Inverse Reinforcement Learning Algorithm for Semi-Markov Decision Processes

Partially Observable Markov Decision Processes and Performance Sensitivity Analysis

Finding Optimal Memoryless Policies of POMDPs under the Expected Average Reward Criterion

Constrained Semi-Markov Decision Processes with Ratio and Time Expected Average Criteria in Polish Spaces

Average Reward Reinforcement Learning For Semi-Markov Decision Processes

Markov Decision Processes under Risk Sensitivity: A Discount Vanishing Approach

Risk-sensitive Semi-Markov Decision Processes with General Utilities and Multiple Criteria