Abstract:Reinforcement learning (RL) has been successfully applied to solve the problem of finding obstacle-free paths for autonomous agents operating in stochastic and uncertain environments. However, when the underlying stochastic dynamics of the environment experiences drastic distribution shifts, the optimal policy obtained in the trained environment may be sub-optimal or may entirely fail in helping find goal-reaching paths for the agent. Approaches like domain randomization and robust RL can provide robust policies, but typically assume minor (bounded) distribution shifts. For substantial distribution shifts, retraining (either with a warm-start policy or from scratch) is an alternative approach. In this paper, we develop a novel approach called {\em Evolutionary Robust Policy Optimization} (ERPO), an adaptive re-training algorithm inspired by evolutionary game theory (EGT). ERPO learns an optimal policy for the shifted environment iteratively using a temperature parameter that controls the trade off between exploration and adherence to the old optimal policy. The policy update itself is an instantiation of the replicator dynamics used in EGT. We show that under fairly common sparsity assumptions on rewards in such environments, ERPO converges to the optimal policy in the shifted environment. We empirically demonstrate that for path finding tasks in a number of environments, ERPO outperforms several popular RL and deep RL algorithms (PPO, A3C, DQN) in many scenarios and popular environments. This includes scenarios where the RL algorithms are allowed to train from scratch in the new environment, when they are retrained on the new environment, or when they are used in conjunction with domain randomization. ERPO shows faster policy adaptation, higher average rewards, and reduced computational costs in policy adaptation.

Revisiting Experience Replayable Conditions

Re-attentive experience replay in off-policy reinforcement learning

A New Reinforcement Learning Algorithm Based on Counterfactual Experience Replay

Replay across Experiments: A Natural Extension of Off-Policy RL

ROER: Regularized Optimal Experience Replay

Actor Prioritized Experience Replay

CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithms

Leveraging Efficiency Through Hybrid Prioritized Experience Replay in Door Environment.

Lucid dreaming for experience replay: refreshing past states with the current policy

Revisiting Prioritized Experience Replay: A Value Perspective

Balanced Prioritized Experience Replay in Off-Policy Reinforcement Learning

Frugal Actor-Critic: Sample Efficient Off-Policy Deep Reinforcement Learning Using Unique Experiences

HiER: Highlight Experience Replay for Boosting Off-Policy Reinforcement Learning Agents

Efficient Reinforcement-Learning Control Algorithm Using Experience Reuse

Replay-enhanced Continual Reinforcement Learning

Z-Score Experience Replay in Off-Policy Deep Reinforcement Learning

Survival of the Fittest: Evolutionary Adaptation of Policies for Environmental Shifts

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

Reverse Experience Replay

Soft Hindsight Experience Replay

A Tighter Convergence Proof of Reverse Experience Replay