Abstract:This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results for model-free DRL-based methods in power systems control problems. But in power systems applications, these model-free methods have certain issues related to training time (clock time) and sample efficiency; both are critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. It is also desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, the state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based DRL framework where a deep neural network (DNN)-based dynamic surrogate model (SM), instead of a real-world power grid or physics-based simulation, is utilized within the policy learning framework, making the process faster and more sample efficient. However, having stable training in model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We addressed these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step loss in surrogate model training. Finally, we achieved 97.5% reduction in samples and 87.7% reduction in training time for an application to the IEEE 300-bus test system.

DistFlow Safe Reinforcement Learning Algorithm for Voltage Magnitude Regulation in Distribution Networks

Safe Imitation Learning-based Optimal Energy Storage Systems Dispatch in Distribution Networks

Robust Offline Deep Reinforcement Learning for Volt-Var Control in Active Distribution Networks

Efficient Learning of Voltage Control Strategies via Model-based Deep Reinforcement Learning

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Enhancing the Tolerance of Voltage Regulation to Cyber Contingencies Via Graph-Based Deep Reinforcement Learning

Augmented Lagrangian-Based Safe Reinforcement Learning Approach for Distribution System Volt/VAR Control

SAVER: Safe Learning-Based Controller for Real-Time Voltage Regulation

Physics-Informed Graphical Representation-Enabled Deep Reinforcement Learning for Robust Distribution System Voltage Control

Deep Reinforcement Learning Based Voltage Control Revisited

Model-Free Voltage Regulation of Unbalanced Distribution Network Based on Surrogate Model and Deep Reinforcement Learning

Robust Deep Reinforcement Learning for Volt-VAR Optimization in Active Distribution System under Uncertainty

Visibility-enhanced Model-Free Deep Reinforcement Learning Algorithm for Voltage Control in Realistic Distribution Systems Using Smart Inverters

Deep Reinforcement Leaming for Short-term Voltage Control by Dynamic Load Shedding in China Southem Power Grid

Deep Reinforcement Learning for Distribution System Cyber Attack Defense with DERs.

Two-Stage Deep Reinforcement Learning for Distribution System Voltage Regulation and Peak Demand Management

A Critical Review of Safe Reinforcement Learning Techniques in Smart Grid Applications

A Deep Reinforcement Learning Voltage Control Method for Distribution Network

Safe Reinforcement Learning for Grid Voltage Control

Deep Reinforcement Learning for Charging Scheduling of Electric Vehicles Considering Distribution Network Voltage Stability

Deep Reinforcement Learning for Long-Term Voltage Stability Control