Abstract:Robust adversarial reinforcement learning is an effective method to train agents to manage uncertain disturbance and modeling errors in real environments. However, for systems that are sensitive to disturbances or those that are difficult to stabilize, it is easier to learn a powerful adversary than establish a stable control policy. An improper strong adversary can destabilize the system, introduce biases in the sampling process, make the learning process unstable, and even reduce the robustness of the policy. In this study, we consider the problem of ensuring system stability during training in the adversarial reinforcement learning architecture. The dissipative principle of robust H-inﬁnity control is extended to the Markov Decision Process, and robust stability constraints are obtained based on L2 gain performance in the reinforcement learning system. Thus, we propose a dissipation-inequation-constraint-based adversarial reinforcement learning architecture. This architecture ensures the stability of the system during training by imposing constraints on the normal and adversarial agents. Theoretically, this architecture can be applied to a large family of deep reinforcement learning algorithms. Results of experiments in MuJoCo and GymFc environments show that our architecture effectively improves the robustness of the controller against environmental changes and adapts to more powerful adversaries. Results of the flight experiments on a real quadcopter indicate that our method can directly deploy the policy trained in the simulation environment to the real environment, and our controller outperforms the PID controller based on hardware-in-the-loop. Both our theoretical and empirical results provide new and critical outlooks on the adversarial reinforcement learning architecture from a rigorous robust control perspective.

A Multiagent Deep Reinforcement Learning-Enabled Dual-Branch Damping Controller for Multimode Oscillation

Robust Damping Controller Design in Renewable- Integrated Power System Considering Communication Resilience

A Novel Deep Reinforcement Learning Enabled Multi-Band PSS for Multi-Mode Oscillation Control

Deep Reinforcement Learning Enabled Bi-Level Robust Parameter Optimization of Hydropower-Dominated Systems for Damping Ultra-Low Frequency Oscillation

Wide-area Time-delay Robust Damping Control Based on System Identification

Wide Area Measurement System-based Low Frequency Oscillation Damping Control through Reinforcement Learning

Design of Damping Controller for Inter-Area Low-Frequency Oscillation Using Time-Delay Wide-Area Measurements

Deep Reinforcement Learning-Based Approach for Proportional Resonance Power System Stabilizer to Prevent Ultra-Low-Frequency Oscillations

Robust Adversarial Reinforcement Learning with Dissipation Inequation Constraint

Hierarchical Control of Multi-Agent Systems using Online Reinforcement Learning

Decentralized Model-Reference Adaptive Control Based Algorithm for Power Systems Inter-Area Oscillation Damping

Adaptive Multi-Surface Sliding Mode Control with Radial Basis Function Neural Networks and Reinforcement Learning for Multirotor Slung Load Systems

SMES Damping Controller Design and Real-Time Parameters Tuning for Low-Frequency Oscillation

Lyapunov-Based Reinforcement Learning for Decentralized Multi-Agent Control

A Multiagent Reinforcement Learning Approach for Wind Farm Frequency Control

A data-driven output voltage control of solid oxide fuel cell using multi-agent deep reinforcement learning

Learning-Based Model Predictive Control of DC-DC Buck Converters in DC Microgrids: A Multi-Agent Deep Reinforcement Learning Approach

Reinforcement learning for multi-agent formation navigation with scalability

Distributed Inter-Area Oscillation Damping Control for Power Systems by Using Wind Generators and Load Aggregators

SODO Based Reinforcement Learning Anti-Disturbance Fault Toler-Ant Control for a Class of Nonlinear Uncertain Systems With Matched and Mismatched Disturbances

Parameter Optimization of Multiple Resonant Controller: A Deep Reinforcement Learning Approach