Abstract:Reinforcement learning (RL) agents are vulnerable to adversarial disturbances, which can deteriorate task performance or break down safety specifications. Existing methods either address safety requirements under the assumption of no adversary (e.g., safe RL) or only focus on robustness against performance adversaries (e.g., robust RL). Learning one policy that is both safe and robust under any adversaries remains a challenging open problem. The difficulty is how to tackle two intertwined aspects in the worst cases: feasibility and optimality. The optimality is only valid inside a feasible region (i.e., robust invariant set), while the identification of maximal feasible region must rely on how to learn the optimal policy. To address this issue, we propose a systematic framework to unify safe RL and robust RL, including the problem formulation, iteration scheme, convergence analysis and practical algorithm design. The unification is built upon constrained two-player zero-sum Markov games, in which the objective for protagonist is twofold. For states inside the maximal robust invariant set, the goal is to pursue rewards under the condition of guaranteed safety; for states outside the maximal robust invariant set, the goal is to reduce the extent of constraint violation. A dual policy iteration scheme is proposed, which simultaneously optimizes a task policy and a safety policy. We prove that the iteration scheme converges to the optimal task policy which maximizes the twofold objective in the worst cases, and the optimal safety policy which stays as far away from the safety boundary. The convergence of safety policy is established by exploiting the monotone contraction property of safety self-consistency operators, and that of task policy depends on the transformation of safety constraints into state-dependent action spaces. By adding two adversarial networks (one is for safety guarantee and the other is for task performance), we propose a practical deep RL algorithm for constrained zero-sum Markov games, called dually robust actor-critic (DRAC). The evaluations with safety-critical benchmarks demonstrate that DRAC achieves high performance and persistent safety under all scenarios (no adversary, safety adversary, performance adversary), outperforming all baselines by a large margin.

Trustworthy autonomous driving via defense-aware robust reinforcement learning against worst-case observational perturbations

Towards Safe Autonomous Driving: Decision Making with Observation-Robust Reinforcement Learning

Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learning

Perceptual Interaction-Based Path Tracking Control of Autonomous Vehicles under DoS Attacks: A Reinforcement Learning Approach

Towards Robust Decision-Making for Autonomous Driving on Highway

Off-Policy Learning-Based Following Control of Cooperative Autonomous Vehicles Under Distributed Attacks

Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies

Trustworthy safety improvement for autonomous driving using reinforcement learning

Towards Robust Decision-Making for Autonomous Highway Driving Based on Safe Reinforcement Learning

Safe Reinforcement Learning with Dual Robustness

Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints

Robustifying Reinforcement Learning Agents via Action Space Adversarial Training

Robust Safe Reinforcement Learning under Adversarial Disturbances

Less is More: A Stealthy and Efficient Adversarial Attack Method for DRL-based Autonomous Driving Policies

Identify, Estimate and Bound the Uncertainty of Reinforcement Learning for Autonomous Driving

Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies

Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement

Enhancing System-Level Safety in Mixed-Autonomy Platoon via Safe Reinforcement Learning

Falsification-Based Robust Adversarial Reinforcement Learning

Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning

Attacking and Defending Deep Reinforcement Learning Policies