Abstract:Reinforcement learning (RL) agents are vulnerable to adversarial disturbances, which can deteriorate task performance or break down safety specifications. Existing methods either address safety requirements under the assumption of no adversary (e.g., safe RL) or only focus on robustness against performance adversaries (e.g., robust RL). Learning one policy that is both safe and robust under any adversaries remains a challenging open problem. The difficulty is how to tackle two intertwined aspects in the worst cases: feasibility and optimality. The optimality is only valid inside a feasible region (i.e., robust invariant set), while the identification of maximal feasible region must rely on how to learn the optimal policy. To address this issue, we propose a systematic framework to unify safe RL and robust RL, including the problem formulation, iteration scheme, convergence analysis and practical algorithm design. The unification is built upon constrained two-player zero-sum Markov games, in which the objective for protagonist is twofold. For states inside the maximal robust invariant set, the goal is to pursue rewards under the condition of guaranteed safety; for states outside the maximal robust invariant set, the goal is to reduce the extent of constraint violation. A dual policy iteration scheme is proposed, which simultaneously optimizes a task policy and a safety policy. We prove that the iteration scheme converges to the optimal task policy which maximizes the twofold objective in the worst cases, and the optimal safety policy which stays as far away from the safety boundary. The convergence of safety policy is established by exploiting the monotone contraction property of safety self-consistency operators, and that of task policy depends on the transformation of safety constraints into state-dependent action spaces. By adding two adversarial networks (one is for safety guarantee and the other is for task performance), we propose a practical deep RL algorithm for constrained zero-sum Markov games, called dually robust actor-critic (DRAC). The evaluations with safety-critical benchmarks demonstrate that DRAC achieves high performance and persistent safety under all scenarios (no adversary, safety adversary, performance adversary), outperforming all baselines by a large margin.

On Practical Robust Reinforcement Learning: Adjacent Uncertainty Set and Double-Agent Algorithm.

Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula

Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training

Robust Reinforcement Learning for Continuous Control with Model Misspecification

Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty

Robust Offline Reinforcement Learning for Non-Markovian Decision Processes

Safe Reinforcement Learning with Dual Robustness

Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization

Robust Multi-Agent Reinforcement Learning with State Uncertainty

Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation

Robust Reinforcement Learning using Offline Data

Robust Proximal Adversarial Reinforcement Learning under Model Mismatch

Robust Reinforcement Learning: A Review of Foundations and Recent Advances

Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Model

Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithm

Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations

Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space

Learning Robust Policies via Interpretable Hamilton-Jacobi Reachability-Guided Disturbances

Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage

Robust Cooperative Multi-Agent Reinforcement Learning:A Mean-Field Type Game Perspective

Time-Constrained Robust MDPs