Abstract:This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results for model-free DRL-based methods in power systems control problems. But in power systems applications, these model-free methods have certain issues related to training time (clock time) and sample efficiency; both are critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. It is also desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, the state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based DRL framework where a deep neural network (DNN)-based dynamic surrogate model (SM), instead of a real-world power grid or physics-based simulation, is utilized within the policy learning framework, making the process faster and more sample efficient. However, having stable training in model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We addressed these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step loss in surrogate model training. Finally, we achieved 97.5% reduction in samples and 87.7% reduction in training time for an application to the IEEE 300-bus test system.

Deep Reinforcement Learning Based Approach for Active Power Security Correction Control of Power System

Exploring the Vulnerability of Deep Reinforcement Learning-based Emergency Control for Low Carbon Power Systems

Adaptive Power System Emergency Control using Deep Reinforcement Learning

Data-driven active corrective control in power systems: an interpretable deep reinforcement learning approach

Active power correction strategies based on deep reinforcement learning-part I: A simulation-driven solution for robustness

A prior knowledge-embedded reinforcement learning method for real-time active power corrective control in complex power systems

A Deep Reinforcement Learning Based Approach for Optimal Active Power Dispatch

Deep Reinforcement Learning for Power System Applications: An Overview

Distributed Deep Reinforcement Learning-based Approach for Fast Preventive Control Considering Transient Stability Constraints

Application and Development of Reinforcement Learning Theory in Power Systems

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Real-Time Optimal Power Flow Method Via Safe Deep Reinforcement Learning Based on Primal-Dual and Prior Knowledge Guidance

Double DQN-based Power System Transient Stability Emergency Control with Protection Coordinations

Make Safe Decisions in Power System: Safe Reinforcement Learning Based Pre-decision Making for Voltage Stability Emergency Control

Deep Forest Reinforcement Learning for Preventive Strategy Considering Automatic Generation Control in Large-Scale Interconnected Power Systems

Power System Emergency Control to Improve Short-Term Voltage Stability Using Deep Reinforcement Learning Algorithm

On Training Effective Reinforcement Learning Agents for Real-time Power Grid Operation and Control

On Fast-Converged Deep Reinforcement Learning for Optimal Dispatch of Large-Scale Power Systems under Transient Security Constraints

Residual Physics and Post-Posed Shielding for Safe Deep Reinforcement Learning Method

Toward Online Power System Model Identification: A Deep Reinforcement Learning Approach

A Deep Reinforcement Learning Framework for Automatic Operation Control of Power System Considering Extreme Weather Events