Abstract:Reinforcement Learning (RL) has gained substantial attention across diverse application domains and theoretical investigations. Existing literature on RL theory largely focuses on risk-neutral settings where the decision-maker learns to maximize the expected cumulative reward. However, in practical scenarios such as portfolio management and e-commerce recommendations, decision-makers often persist in heterogeneous risk preferences subject to outcome uncertainties, which can not be well-captured by the risk-neural framework. Incorporating these preferences can be approached through utility theory, yet the development of risk-sensitive RL under general utility functions remains an open question for theoretical exploration. In this paper, we consider a scenario where the decision-maker seeks to optimize a general utility function of the cumulative reward in the framework of a Markov decision process (MDP). To facilitate the Dynamic Programming Principle and Bellman equation, we enlarge the state space with an additional dimension that accounts for the cumulative reward. We propose a discretized approximation scheme to the MDP under enlarged state space, which is tractable and key for algorithmic design. We then propose a modified value iteration algorithm that employs an epsilon-covering over the space of cumulative reward. When a simulator is accessible, our algorithm efficiently learns a near-optimal policy with guaranteed sample complexity. In the absence of a simulator, our algorithm, designed with an upper-confidence-bound exploration approach, identifies a near-optimal policy while ensuring a guaranteed regret bound. For both algorithms, we match the theoretical lower bounds for the risk-neutral setting.

Epistemic Risk-Sensitive Reinforcement Learning

Risk-sensitive Markov Decision Process and Learning under General Utility Functions

Risk-Sensitive and Robust Model-Based Reinforcement Learning and Planning

RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Risk

Risk-Sensitive Reinforcement Learning with Exponential Criteria

Risk-Sensitive Bayesian Games for Multi-Agent Reinforcement Learning under Policy Uncertainty

Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning

Risk-Sensitive Reinforcement Learning via Policy Gradient Search

One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning

Uncertainty-Aware Reinforcement Learning for Portfolio Optimization

Risk Preferences of Learning Algorithms

Risk-Averse Reinforcement Learning via Dynamic Time-Consistent Risk Measures

Risk-Sensitive Reinforcement Learning: a Martingale Approach to Reward Uncertainty

Reinforcement Learning with Dynamic Convex Risk Measures

Robust Reinforcement Learning with Dynamic Distortion Risk Measures

Fundamental Limits of Reinforcement Learning in Environment with Endogeneous and Exogeneous Uncertainty

Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning

RA-RLHF: Provably Efficient Risk-Aware Reinforcement Learning Human Feedback

Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition

Risk-sensitive Inverse Reinforcement Learning via Semi- and Non-Parametric Methods

Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis