Abstract:This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results for model-free DRL-based methods in power systems control problems. But in power systems applications, these model-free methods have certain issues related to training time (clock time) and sample efficiency; both are critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. It is also desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, the state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based DRL framework where a deep neural network (DNN)-based dynamic surrogate model (SM), instead of a real-world power grid or physics-based simulation, is utilized within the policy learning framework, making the process faster and more sample efficient. However, having stable training in model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We addressed these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step loss in surrogate model training. Finally, we achieved 97.5% reduction in samples and 87.7% reduction in training time for an application to the IEEE 300-bus test system.

An Online Deep Reinforcement Learning Based Parameter Identification Method for HVDC System

Toward Online Power System Model Identification: A Deep Reinforcement Learning Approach

Application Of Recursive Least Squares Method For Parameter Identification Of Deh Governing System

Recursive Least Square Method Based Parameter Identification of Steam Turbine Digital Electric-Hydraulic Governing System

An Intelligent Parameter Identification Method of DFIG Systems Using Hybrid Particle Swarm Optimization and Reinforcement Learning

Deep Reinforcement Learning Based Approach for Active Power Security Correction Control of Power System

Reinforcement Learning based Method for Online Parameter Identification of the Permanent Magnet Synchronous Machines

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Efficient Learning of Voltage Control Strategies via Model-based Deep Reinforcement Learning

Fault Identification in Power Network Based on Deep Reinforcement Learning

Simplified Deep Reinforcement Learning Based Volt-var Control of Topologically Variable Power System

A Deep Reinforcement Learning Based Approach for Optimal Active Power Dispatch

A Deep Reinforcement Learning Design for Virtual Synchronous Generators Accommodating Modular Multilevel Converters

Research on the Identification of Dominant Dynamic Parameters of Power System

An Online Identification Method for Health State Parameters of Thyristor Modules in HVDC Converter Valve

Deep Reinforcement Learning Control for Pulsed Power Load Online Deployment in DC Shipboard Integrated Power System

Hierarchical deep reinforcement learning for self-adaptive economic dispatch

AUV Hydrodynamic Coefficient Offline Identification Based on Deep Reinforcement Learning

Adaptive Power System Emergency Control using Deep Reinforcement Learning

Parameter Identification of Doubly-fed Wind Farm Aggregated Model with Reinforcement Learning.

Combination optimization method of grid sections based on deep reinforcement learning with accelerated convergence speed