Abstract:This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results for model-free DRL-based methods in power systems control problems. But in power systems applications, these model-free methods have certain issues related to training time (clock time) and sample efficiency; both are critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. It is also desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, the state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based DRL framework where a deep neural network (DNN)-based dynamic surrogate model (SM), instead of a real-world power grid or physics-based simulation, is utilized within the policy learning framework, making the process faster and more sample efficient. However, having stable training in model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We addressed these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step loss in surrogate model training. Finally, we achieved 97.5% reduction in samples and 87.7% reduction in training time for an application to the IEEE 300-bus test system.

Data-Driven Passivity Analysis and Fault Detection Using Reinforcement Learning

Exploring the Vulnerability of Deep Reinforcement Learning-based Emergency Control for Low Carbon Power Systems

Data-driven active corrective control in power systems: an interpretable deep reinforcement learning approach

A Reinforcement Learning System for Fault Detection and Diagnosis in Mechatronic Systems

Deep reinforcement learning based home energy management system with devices operational dependencies

Residual Physics and Post-Posed Shielding for Safe Deep Reinforcement Learning Method

Safe Reinforcement Learning for Power System Control: A Review

Integrating reinforcement learning with deterministic learning for fault diagnosis of nonlinear systems

A Deep Reinforcement Learning Based Approach for Optimal Active Power Dispatch

Automated Model Generation for Machinery Fault Diagnosis Based on Reinforcement Learning and Neural Architecture Search

Adaptive Power System Emergency Control using Deep Reinforcement Learning

Safe Reinforcement Learning for Emergency LoadShedding of Power Systems

A Reinforcement Learning Engine with Reduced Action and State Space for Scalable Cyber-Physical Optimal Response

Data-Driven Fault Detection for Lithium-Ion Battery Packs Via Behavioral System Representation

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Reinforcement Learning-Based Optimal Fault-Tolerant Tracking Control of Industrial Processes

Online Operational Decision-making for Integrated Electric-Gas Systems with Safe Reinforcement Learning

Analyzing Robustness of the Deep Reinforcement Learning Algorithm in Ramp Metering Applications Considering False Data Injection Attack and Defense

Online Cyber-Attack Detection in the Industrial Control System: A Deep Reinforcement Learning Approach

Real-Time Energy Management in Smart Homes Through Deep Reinforcement Learning

An Online Search Method for Representative Risky Fault Chains Based on Reinforcement Learning and Knowledge Transfer