Abstract:Robust adversarial reinforcement learning is an effective method to train agents to manage uncertain disturbance and modeling errors in real environments. However, for systems that are sensitive to disturbances or those that are difficult to stabilize, it is easier to learn a powerful adversary than establish a stable control policy. An improper strong adversary can destabilize the system, introduce biases in the sampling process, make the learning process unstable, and even reduce the robustness of the policy. In this study, we consider the problem of ensuring system stability during training in the adversarial reinforcement learning architecture. The dissipative principle of robust H-inﬁnity control is extended to the Markov Decision Process, and robust stability constraints are obtained based on L2 gain performance in the reinforcement learning system. Thus, we propose a dissipation-inequation-constraint-based adversarial reinforcement learning architecture. This architecture ensures the stability of the system during training by imposing constraints on the normal and adversarial agents. Theoretically, this architecture can be applied to a large family of deep reinforcement learning algorithms. Results of experiments in MuJoCo and GymFc environments show that our architecture effectively improves the robustness of the controller against environmental changes and adapts to more powerful adversaries. Results of the flight experiments on a real quadcopter indicate that our method can directly deploy the policy trained in the simulation environment to the real environment, and our controller outperforms the PID controller based on hardware-in-the-loop. Both our theoretical and empirical results provide new and critical outlooks on the adversarial reinforcement learning architecture from a rigorous robust control perspective.

Adversarial Skill Learning for Robust Manipulation

Learning Robust Skills for Tightly Coordinated Arms in Contact-Rich Tasks

Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space

Learning Robust Policies via Interpretable Hamilton-Jacobi Reachability-Guided Disturbances

Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisition

Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization

Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learning

Continuous control actions learning and adaptation for robotic manipulation through reinforcement learning

Curriculum Adversarial Training for Robust Reinforcement Learning

Robust Safe Reinforcement Learning under Adversarial Disturbances

Robust Manipulation Primitive Learning via Domain Contraction

Co-Designing Tools and Control Policies for Robust Manipulation

Fault-Aware Robust Control via Adversarial Reinforcement Learning

A Robust Mean-Field Actor-Critic Reinforcement Learning Against Adversarial Perturbations on Agent States

Robustifying Reinforcement Learning Agents via Action Space Adversarial Training

Adversarial Policies: Attacking Deep Reinforcement Learning

Robust Adaptive Ensemble Adversary Reinforcement Learning

Robotic Control in Adversarial and Sparse Reward Environments: A Robust Goal-Conditioned Reinforcement Learning Approach

Robust Adversarial Reinforcement Learning with Dissipation Inequation Constraint

A High-Efficient Reinforcement Learning Approach for Dexterous Manipulation

Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations