Abstract:Reinforcement learning (RL) agents are vulnerable to adversarial disturbances, which can deteriorate task performance or break down safety specifications. Existing methods either address safety requirements under the assumption of no adversary (e.g., safe RL) or only focus on robustness against performance adversaries (e.g., robust RL). Learning one policy that is both safe and robust under any adversaries remains a challenging open problem. The difficulty is how to tackle two intertwined aspects in the worst cases: feasibility and optimality. The optimality is only valid inside a feasible region (i.e., robust invariant set), while the identification of maximal feasible region must rely on how to learn the optimal policy. To address this issue, we propose a systematic framework to unify safe RL and robust RL, including the problem formulation, iteration scheme, convergence analysis and practical algorithm design. The unification is built upon constrained two-player zero-sum Markov games, in which the objective for protagonist is twofold. For states inside the maximal robust invariant set, the goal is to pursue rewards under the condition of guaranteed safety; for states outside the maximal robust invariant set, the goal is to reduce the extent of constraint violation. A dual policy iteration scheme is proposed, which simultaneously optimizes a task policy and a safety policy. We prove that the iteration scheme converges to the optimal task policy which maximizes the twofold objective in the worst cases, and the optimal safety policy which stays as far away from the safety boundary. The convergence of safety policy is established by exploiting the monotone contraction property of safety self-consistency operators, and that of task policy depends on the transformation of safety constraints into state-dependent action spaces. By adding two adversarial networks (one is for safety guarantee and the other is for task performance), we propose a practical deep RL algorithm for constrained zero-sum Markov games, called dually robust actor-critic (DRAC). The evaluations with safety-critical benchmarks demonstrate that DRAC achieves high performance and persistent safety under all scenarios (no adversary, safety adversary, performance adversary), outperforming all baselines by a large margin.

Improving the Robustness of Reinforcement Learning Policies with $\mathcal{L}_{1}$ Adaptive Control

Robust Model Based Reinforcement Learning Using $\mathcal{L}_1$ Adaptive Control

H_∞ Model-free Reinforcement Learning with Robust Stability Guarantee

Active Robust Adversarial Reinforcement Learning under Temporally-Coupled Perturbations

Robust Reinforcement Learning for Continuous Control with Model Misspecification

Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space

Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies

Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems

Robust Safe Reinforcement Learning under Adversarial Disturbances

Robust Adaptive Ensemble Adversary Reinforcement Learning

Robust Control for Affine Nonlinear Systems under the Reinforcement Learning Framework

Robustifying Reinforcement Learning Agents via Action Space Adversarial Training

Data-Driven Robust Control of Discrete-Time Uncertain Linear Systems Via Off-Policy Reinforcement Learning.

Robust Proximal Adversarial Reinforcement Learning under Model Mismatch

Off-Policy Risk-Sensitive Reinforcement Learning-Based Constrained Robust Optimal Control

Risk Averse Robust Adversarial Reinforcement Learning

Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learning

Learning Robust Policies via Interpretable Hamilton-Jacobi Reachability-Guided Disturbances

Robust Control of Uncertain Linear Systems Based on Reinforcement Learning Principles.

Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

Safe Reinforcement Learning with Dual Robustness