Abstract:This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results for model-free DRL-based methods in power systems control problems. But in power systems applications, these model-free methods have certain issues related to training time (clock time) and sample efficiency; both are critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. It is also desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, the state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based DRL framework where a deep neural network (DNN)-based dynamic surrogate model (SM), instead of a real-world power grid or physics-based simulation, is utilized within the policy learning framework, making the process faster and more sample efficient. However, having stable training in model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We addressed these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step loss in surrogate model training. Finally, we achieved 97.5% reduction in samples and 87.7% reduction in training time for an application to the IEEE 300-bus test system.

An unsolvable power flow adjustment method for weak power grid based on transmission channel positioning and deep reinforcement learning

Exploring the Vulnerability of Deep Reinforcement Learning-based Emergency Control for Low Carbon Power Systems

Transmission Interface Power Flow Adjustment: A Deep Reinforcement Learning Approach based on Multi-task Attribution Map

A Safe DRL Method for Fast Solution of Real-Time Optimal Power Flow

A Data-driven Method for Fast AC Optimal Power Flow Solutions via Deep Reinforcement Learning

Reactive Power Flow Convergence Adjustment Based on Deep Reinforcement Learning

Power System Flow Adjustment and Sample Generation Based on Deep Reinforcement Learning

Deep Reinforcement Learning-Based Tie-Line Power Adjustment Method for Power System Operation State Calculation

Physics-Informed Reinforcement Learning for Real-Time Optimal Power Flow with Renewable Energy Resources

Power balance control of RES integrated power system by deep reinforcement learning with optimized utilization rate of renewable energy

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

A Deep Reinforcement Learning Based Approach for Optimal Active Power Dispatch

Deep reinforcement learning based power system optimal carbon emission flow

Distributed Deep Reinforcement Learning-based Approach for Fast Preventive Control Considering Transient Stability Constraints

Constrained Reinforcement Learning for Predictive Control in Real-Time Stochastic Dynamic Optimal Power Flow

Combination optimization method of grid sections based on deep reinforcement learning with accelerated convergence speed

Power flow adjustment for smart microgrid based on edge computing and multi-agent deep reinforcement learning

Multi-Stage Transmission Line Flow Control Using Centralized and Decentralized Reinforcement Learning Agents

Optimizing Load Scheduling in Power Grids Using Reinforcement Learning and Markov Decision Processes

Deep Reinforcement Leaming for Short-term Voltage Control by Dynamic Load Shedding in China Southem Power Grid

Real-time Sequential Security-Constrained Optimal Power Flow: A Hybrid Knowledge-Data-Driven Reinforcement Learning Approach