Abstract:Reinforcement learning (RL) agents are vulnerable to adversarial disturbances, which can deteriorate task performance or break down safety specifications. Existing methods either address safety requirements under the assumption of no adversary (e.g., safe RL) or only focus on robustness against performance adversaries (e.g., robust RL). Learning one policy that is both safe and robust under any adversaries remains a challenging open problem. The difficulty is how to tackle two intertwined aspects in the worst cases: feasibility and optimality. The optimality is only valid inside a feasible region (i.e., robust invariant set), while the identification of maximal feasible region must rely on how to learn the optimal policy. To address this issue, we propose a systematic framework to unify safe RL and robust RL, including the problem formulation, iteration scheme, convergence analysis and practical algorithm design. The unification is built upon constrained two-player zero-sum Markov games, in which the objective for protagonist is twofold. For states inside the maximal robust invariant set, the goal is to pursue rewards under the condition of guaranteed safety; for states outside the maximal robust invariant set, the goal is to reduce the extent of constraint violation. A dual policy iteration scheme is proposed, which simultaneously optimizes a task policy and a safety policy. We prove that the iteration scheme converges to the optimal task policy which maximizes the twofold objective in the worst cases, and the optimal safety policy which stays as far away from the safety boundary. The convergence of safety policy is established by exploiting the monotone contraction property of safety self-consistency operators, and that of task policy depends on the transformation of safety constraints into state-dependent action spaces. By adding two adversarial networks (one is for safety guarantee and the other is for task performance), we propose a practical deep RL algorithm for constrained zero-sum Markov games, called dually robust actor-critic (DRAC). The evaluations with safety-critical benchmarks demonstrate that DRAC achieves high performance and persistent safety under all scenarios (no adversary, safety adversary, performance adversary), outperforming all baselines by a large margin.

Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales

Robust Reinforcement Learning from Corrupted Human Feedback

Adaptive Dense Reward: Understanding the Gap Between Action and Reward Space in Alignment

Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning

Reward-Robust RLHF in LLMs

Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula

Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Model

Reinforcement Learning with Perturbed Rewards

Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training

Stabilizing RLHF through Advantage Model and Selective Rehearsal

Data-Efficient Deep Reinforcement Learning with Symmetric Consistency

Adaptive Preference Scaling for Reinforcement Learning with Human Feedback

Towards Reliable Alignment: Uncertainty-aware RLHF

Constrained Reinforcement Learning Under Model Mismatch

Optimizing Language Models with Fair and Stable Reward Composition in Reinforcement Learning

Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity

RLSF: Reinforcement Learning via Symbolic Feedback

Teaching Large Language Models to Reason with Reinforcement Learning

Confronting Reward Model Overoptimization with Constrained RLHF

Safe Reinforcement Learning with Dual Robustness

Robust Reinforcement Learning for Continuous Control with Model Misspecification