Abstract:Safety is a primary concern when applying reinforcement learning to real-world control tasks, especially in the presence of external disturbances. However, existing safe reinforcement learning algorithms rarely account for external disturbances, limiting their applicability and robustness in practice. To address this challenge, this paper proposes a robust safe reinforcement learning framework that tackles worst-case disturbances. First, this paper presents a policy iteration scheme to solve for the robust invariant set, i.e., a subset of the safe set, where persistent safety is only possible for states within. The key idea is to establish a two-player zero-sum game by leveraging the safety value function in Hamilton-Jacobi reachability analysis, in which the protagonist (i.e., control inputs) aims to maintain safety and the adversary (i.e., external disturbances) tries to break down safety. This paper proves that the proposed policy iteration algorithm converges monotonically to the maximal robust invariant set. Second, this paper integrates the proposed policy iteration scheme into a constrained reinforcement learning algorithm that simultaneously synthesizes the robust invariant set and uses it for constrained policy optimization. This algorithm tackles both optimality and safety, i.e., learning a policy that attains high rewards while maintaining safety under worst-case disturbances. Experiments on classic control tasks show that the proposed method achieves zero constraint violation with learned worst-case adversarial disturbances, while other baseline algorithms violate the safety constraints substantially. Our proposed method also attains comparable performance as the baselines even in the absence of the adversary.

What problem does this paper attempt to address?

This paper attempts to solve the safety problem of applying reinforcement learning to practical control tasks in the presence of external disturbances. Specifically, existing safe reinforcement learning algorithms rarely consider external disturbances, which limits their applicability and robustness in practice. To address this challenge, this paper proposes a robust safe reinforcement learning framework aimed at handling worst - case disturbances. ### Main Contributions 1. **Proposed a policy iteration scheme**: This scheme is used to synthesize the maximum robust invariant set. The robust invariant set refers to the set of states that can remain safe continuously under the worst - case disturbance. By using the safety value function in Hamilton - Jacobi reachability analysis, a two - player zero - sum game model is established, where the protagonist (i.e., the control input) aims to maintain safety, while the antagonist (i.e., the external disturbance) attempts to break safety. This paper proves that the proposed policy iteration algorithm monotonically converges to the maximum robust invariant set. 2. **Designed a constrained reinforcement learning algorithm**: This algorithm simultaneously synthesizes the robust invariant set and uses it for constrained policy optimization. The algorithm not only pursues optimal performance but also ensures safety under the worst - case disturbance. Experimental results show that the proposed method achieves zero constraint violations in classical control tasks, while other baseline algorithms severely violate safety constraints under the worst - case disturbance. ### Experimental Verification - **Double - integrator experiment**: Verified the effectiveness of the proposed policy iteration scheme in learning the maximum robust invariant set. The results show that the learned robust invariant set is very consistent with the true invariant set, and in the presence of disturbances, the robust invariant set is indeed smaller than the standard invariant set. - **Cart - pole and quadrotor tasks**: Tested the proposed SAC - RIS algorithm and other baseline algorithms in two safety - critical control tasks. Experimental results show that the SAC - RIS algorithm not only achieves high performance under the worst - case disturbance but also completely avoids violating safety constraints. Even in the absence of disturbances, the performance of the SAC - RIS algorithm is comparable to that of the baseline algorithms, and no safety constraints are violated. ### Conclusion This paper proposes a robust safe reinforcement learning framework that can learn optimal safety policies in the presence of external disturbances. Through the policy iteration scheme and the constrained reinforcement learning algorithm, this paper effectively addresses the shortcomings of existing safe reinforcement learning algorithms when facing external disturbances. Experimental results verify the effectiveness of this framework. ### Limitations Although the proposed algorithm performs well in the simulation environment, there are still some constraint violations during the training process. Therefore, this algorithm is not currently suitable for direct online deployment and needs to be applied to the actual system after training in the simulation environment.

Robust Safe Reinforcement Learning under Adversarial Disturbances

Safe Reinforcement Learning with Dual Robustness

On the Robustness of Safe Reinforcement Learning under Observational Perturbations

Learning Robust Policies via Interpretable Hamilton-Jacobi Reachability-Guided Disturbances

Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learning

Robust Reinforcement Learning with UUB Guarantee for Safe Motion Control of Autonomous Robots

Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learning

Towards Robust and Safe Reinforcement Learning with Benign Off-policy Data.

Safe Reinforcement Learning Using Robust Control Barrier Functions

Robustifying Reinforcement Learning Agents via Action Space Adversarial Training

ROSCOM: Robust Safe Reinforcement Learning on Stochastic Constraint Manifolds

Safety Robustness of Reinforcement Learning Policies: A View from Robust Control

Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guarantees

Robustifying Reinforcement Learning Policies with L1 Adaptive Control

Robust Adversarial Reinforcement Learning with Dissipation Inequation Constraint

Risk Averse Robust Adversarial Reinforcement Learning

ISAACS: Iterative Soft Adversarial Actor-Critic for Safety

Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards

Certifying Safety in Reinforcement Learning under Adversarial Perturbation Attacks

H_∞ Model-free Reinforcement Learning with Robust Stability Guarantee

Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space