Abstract:Reinforcement learning (RL) has been successfully applied to solve the problem of finding obstacle-free paths for autonomous agents operating in stochastic and uncertain environments. However, when the underlying stochastic dynamics of the environment experiences drastic distribution shifts, the optimal policy obtained in the trained environment may be sub-optimal or may entirely fail in helping find goal-reaching paths for the agent. Approaches like domain randomization and robust RL can provide robust policies, but typically assume minor (bounded) distribution shifts. For substantial distribution shifts, retraining (either with a warm-start policy or from scratch) is an alternative approach. In this paper, we develop a novel approach called {\em Evolutionary Robust Policy Optimization} (ERPO), an adaptive re-training algorithm inspired by evolutionary game theory (EGT). ERPO learns an optimal policy for the shifted environment iteratively using a temperature parameter that controls the trade off between exploration and adherence to the old optimal policy. The policy update itself is an instantiation of the replicator dynamics used in EGT. We show that under fairly common sparsity assumptions on rewards in such environments, ERPO converges to the optimal policy in the shifted environment. We empirically demonstrate that for path finding tasks in a number of environments, ERPO outperforms several popular RL and deep RL algorithms (PPO, A3C, DQN) in many scenarios and popular environments. This includes scenarios where the RL algorithms are allowed to train from scratch in the new environment, when they are retrained on the new environment, or when they are used in conjunction with domain randomization. ERPO shows faster policy adaptation, higher average rewards, and reduced computational costs in policy adaptation.

Reinforcement learning with non-ergodic reward increments: robustness via ergodicity transformations

Fundamental Limits of Reinforcement Learning in Environment with Endogeneous and Exogeneous Uncertainty

Robust Reinforcement Learning with Dynamic Distortion Risk Measures

Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach

Robust Reinforcement Learning: A Review of Foundations and Recent Advances

Deep Reinforcement Trading with Predictable Returns

Reinforcement learning algorithm for non-stationary environments

Survival of the Fittest: Evolutionary Adaptation of Policies for Environmental Shifts

Reward Shaping via Diffusion Process in Reinforcement Learning

ELO-Rated Sequence Rewards: Advancing Reinforcement Learning Models

DROP: Distributional and Regular Optimism and Pessimism for Reinforcement Learning

Reinforcement Learning with Perturbed Rewards

NeoRL: Efficient Exploration for Nonepisodic RL

Predictable Reinforcement Learning Dynamics through Entropy Rate Minimization

Robust Reinforcement Learning under Diffusion Models for Data with Jumps

Single-Trajectory Distributionally Robust Reinforcement Learning

Learning Optimal Advantage from Preferences and Mistaking it for Reward

Robot See, Robot Do: Imitation Reward for Noisy Financial Environments

Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales

Learning Long-Term Reward Redistribution via Randomized Return Decomposition

Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty