Abstract:This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results for model-free DRL-based methods in power systems control problems. But in power systems applications, these model-free methods have certain issues related to training time (clock time) and sample efficiency; both are critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. It is also desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, the state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based DRL framework where a deep neural network (DNN)-based dynamic surrogate model (SM), instead of a real-world power grid or physics-based simulation, is utilized within the policy learning framework, making the process faster and more sample efficient. However, having stable training in model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We addressed these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step loss in surrogate model training. Finally, we achieved 97.5% reduction in samples and 87.7% reduction in training time for an application to the IEEE 300-bus test system.

Deep Reinforcement Learning for Long-Term Voltage Stability Control

Efficient Learning of Voltage Control Strategies via Model-based Deep Reinforcement Learning

Deep Reinforcement Learning for Electric Transmission Voltage Control

Deep Reinforcement Learning Based Voltage Control Revisited

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Deep Reinforcement Leaming for Short-term Voltage Control by Dynamic Load Shedding in China Southem Power Grid

Stability Constrained Reinforcement Learning for Decentralized Real-Time Voltage Control

Power System Emergency Control to Improve Short-Term Voltage Stability Using Deep Reinforcement Learning Algorithm

Load Shedding Scheme with Deep Reinforcement Learning to Improve Short-term Voltage Stability

Simplified Deep Reinforcement Learning Based Volt-var Control of Topologically Variable Power System

Safe Reinforcement Learning for Grid Voltage Control

Implementing Deep Reinforcement Learning-Based Grid Voltage Control in Real-World Power Systems: Challenges and Insights

Scalable Voltage Control using Structure-Driven Hierarchical Deep Reinforcement Learning

Deep Reinforcement Learning for Voltage Control and Renewable Accommodation Using Spatial-Temporal Graph Information

A Date-Driven Voltage Control Strategy for Distribution Network Using Deep Reinforcement Learning

Distributed Deep Reinforcement Learning-based Approach for Fast Preventive Control Considering Transient Stability Constraints

Three-Stage Inverter-Based Peak Shaving and Volt-VAR Control in Active Distribution Networks Using Online Safe Deep Reinforcement Learning

Robust Offline Deep Reinforcement Learning for Volt-Var Control in Active Distribution Networks

Optimal control with deep reinforcement learning for shunt compensations to enhance voltage stability

Safe Reinforcement Learning for Emergency LoadShedding of Power Systems