Abstract:Reinforcement learning has demonstrated impressive performance in various challenging problems such as robotics, board games, and classical arcade games. However, its real-world applications can be hindered by the absence of robustness and safety in the learned policies. More specifically, an RL agent that trains in a certain Markov decision process (MDP) often struggles to perform well in nearly identical MDPs. To address this issue, we employ the framework of Robust MDPs (RMDPs) in a model-based setting and introduce a novel learned transition model. Our method specifically incorporates an auxiliary pessimistic model, updated adversarially, to estimate the worst-case MDP within a Kullback-Leibler uncertainty set. In comparison to several existing works, our work does not impose any additional conditions on the training environment, such as the need for a parametric simulator. To test the effectiveness of the proposed pessimistic model in enhancing policy robustness, we integrate it into a practical RL algorithm, called Robust Model-Based Policy Optimization (RMBPO). Our experimental results indicate a notable improvement in policy robustness on high-dimensional MuJoCo control tasks, with the auxiliary model enhancing the performance of the learned policy in distorted MDPs. We further explore the learned deviation between the proposed auxiliary world model and the nominal model, to examine how pessimism is achieved. By learning a pessimistic world model and demonstrating its role in improving policy robustness, our research contributes towards making (model-based) RL more robust.

Robust Proximal Adversarial Reinforcement Learning under Model Mismatch

Constrained Reinforcement Learning Under Model Mismatch

Robust Adaptive Ensemble Adversary Reinforcement Learning

Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Model

Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space

Online Policy Optimization for Robust MDP

Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies

Active Robust Adversarial Reinforcement Learning under Temporally-Coupled Perturbations

Robust Reinforcement Learning through Efficient Adversarial Herding

Robust Reinforcement Learning for Continuous Control with Model Misspecification

Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

Risk Averse Robust Adversarial Reinforcement Learning

On Practical Robust Reinforcement Learning: Adjacent Uncertainty Set and Double-Agent Algorithm.

Robustifying Reinforcement Learning Policies with L1 Adaptive Control

Online Robust Policy Learning in the Presence of Unknown Adversaries

Robust Reinforcement Learning under Model Misspecification

Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training

Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations

Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games

Partial End-to-end Reinforcement Learning for Robustness Against Modelling Error in Autonomous Racing

Policy Gradient Method For Robust Reinforcement Learning