Abstract:This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results for model-free DRL-based methods in power systems control problems. But in power systems applications, these model-free methods have certain issues related to training time (clock time) and sample efficiency; both are critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. It is also desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, the state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based DRL framework where a deep neural network (DNN)-based dynamic surrogate model (SM), instead of a real-world power grid or physics-based simulation, is utilized within the policy learning framework, making the process faster and more sample efficient. However, having stable training in model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We addressed these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step loss in surrogate model training. Finally, we achieved 97.5% reduction in samples and 87.7% reduction in training time for an application to the IEEE 300-bus test system.

SAVER: Safe Learning-Based Controller for Real-Time Voltage Regulation

AdapSafe: Adaptive and Safe-Certified Deep Reinforcement Learning-Based Frequency Control for Carbon-Neutral Power Systems.

DistFlow Safe Reinforcement Learning Algorithm for Voltage Magnitude Regulation in Distribution Networks

Predictive Control of Voltage Source Inverter: an Online Reinforcement Learning Solution

Three-Stage Inverter-Based Peak Shaving and Volt-VAR Control in Active Distribution Networks Using Online Safe Deep Reinforcement Learning

Safe Imitation Learning-based Optimal Energy Storage Systems Dispatch in Distribution Networks

Efficient Learning of Voltage Control Strategies via Model-based Deep Reinforcement Learning

Safety-Enhanced Self-Learning for Optimal Power Converter Control

Data-Driven Robust Voltage/VAR Control Using PV Inverters in Active Distribution Networks

Two-Stage Deep Reinforcement Learning for Inverter-Based Volt-VAR Control in Active Distribution Networks

Safe Reinforcement Learning for Grid Voltage Control

Stability Constrained Reinforcement Learning for Decentralized Real-Time Voltage Control

Robust Deep Reinforcement Learning for Volt-VAR Optimization in Active Distribution System under Uncertainty

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Model-Free Voltage Regulation of Unbalanced Distribution Network Based on Surrogate Model and Deep Reinforcement Learning

Augmented Lagrangian-Based Safe Reinforcement Learning Approach for Distribution System Volt/VAR Control

Online Volt-Var Control Based on Reinforcement Learning in Active Distribution Systems Considering PV Inverters

Online Voltage Control Strategy: Multi-Mode Based Data-Driven Approach for Active Distribution Networks

Robust Offline Deep Reinforcement Learning for Volt-Var Control in Active Distribution Networks

Make Safe Decisions in Power System: Safe Reinforcement Learning Based Pre-decision Making for Voltage Stability Emergency Control

Two-stage scheduling of smart electric vehicle charging stations and inverter-based Volt-VAR control using a prediction error-integrated deep reinforcement learning method