Abstract:This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results for model-free DRL-based methods in power systems control problems. But in power systems applications, these model-free methods have certain issues related to training time (clock time) and sample efficiency; both are critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. It is also desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, the state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based DRL framework where a deep neural network (DNN)-based dynamic surrogate model (SM), instead of a real-world power grid or physics-based simulation, is utilized within the policy learning framework, making the process faster and more sample efficient. However, having stable training in model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We addressed these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step loss in surrogate model training. Finally, we achieved 97.5% reduction in samples and 87.7% reduction in training time for an application to the IEEE 300-bus test system.

Real-time Autonomous Line Flow Control Using Proximal Policy Optimization

Controlling Pressure of Gas Pipeline Network Based on Mixed Proximal Policy Optimization

Real-Time Optimal Power Flow Method Via Safe Deep Reinforcement Learning Based on Primal-Dual and Prior Knowledge Guidance

A Safe DRL Method for Fast Solution of Real-Time Optimal Power Flow

A Data-driven Method for Fast AC Optimal Power Flow Solutions via Deep Reinforcement Learning

Physics-Informed Reinforcement Learning for Real-Time Optimal Power Flow with Renewable Energy Resources

A Graph-based Deep Reinforcement Learning Framework for Autonomous Power Dispatch on Power Systems with Changing Topologies

Proximal policy optimization through a deep reinforcement learning framework for remedial action schemes of VSC-HVDC

Constrained Reinforcement Learning for Predictive Control in Real-Time Stochastic Dynamic Optimal Power Flow

DDPG-based Reactive Power Optimization Strategy for Active Distribution Network

Action Set Based Policy Optimization for Safe Power Grid Management

Deep Reinforcement Learning Based Approach for Optimal Power Flow of Distribution Networks Embedded with Renewable Energy and Storage Devices

Distributed Deep Reinforcement Learning-based Approach for Fast Preventive Control Considering Transient Stability Constraints

Real-time Sequential Security-Constrained Optimal Power Flow: A Hybrid Knowledge-Data-Driven Reinforcement Learning Approach

On Training Effective Reinforcement Learning Agents for Real-time Power Grid Operation and Control

Safe deep reinforcement learning for real-time AC optimal power flow: A near-optimal solution

Online Preventive Control for Transmission Overload Relief Using Safe Reinforcement Learning with Enhanced Spatial-Temporal Awareness

Optimizing Load Scheduling in Power Grids Using Reinforcement Learning and Markov Decision Processes

A Deep Reinforcement Learning Based Approach for Optimal Active Power Dispatch

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Multi-Stage Transmission Line Flow Control Using Centralized and Decentralized Reinforcement Learning Agents