Abstract:This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results for model-free DRL-based methods in power systems control problems. But in power systems applications, these model-free methods have certain issues related to training time (clock time) and sample efficiency; both are critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. It is also desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, the state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based DRL framework where a deep neural network (DNN)-based dynamic surrogate model (SM), instead of a real-world power grid or physics-based simulation, is utilized within the policy learning framework, making the process faster and more sample efficient. However, having stable training in model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We addressed these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step loss in surrogate model training. Finally, we achieved 97.5% reduction in samples and 87.7% reduction in training time for an application to the IEEE 300-bus test system.

Optimal control with deep reinforcement learning for shunt compensations to enhance voltage stability

Research on Volt/Var Control of Distribution Networks Based on PPO Algorithm

Distributed Deep Reinforcement Learning-based Approach for Fast Preventive Control Considering Transient Stability Constraints

Efficient Learning of Voltage Control Strategies via Model-based Deep Reinforcement Learning

Predictive Control of Voltage Source Inverter: an Online Reinforcement Learning Solution

Residual Deep Reinforcement Learning for Inverter-based Volt-Var Control

Deep Reinforcement Learning for Long-Term Voltage Stability Control

Research on the multi-timescale optimal voltage control method for distribution network based on a DQN-DDPG algorithm

DDPG-based Reactive Power Optimization Strategy for Active Distribution Network

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Robust Federated Deep Reinforcement Learning for Optimal Control in Multiple Virtual Power Plants with Electric Vehicles

Deep-Reinforcement-Learning-Based Two-Timescale Voltage Control for Distribution Systems

Scalable Voltage Control using Structure-Driven Hierarchical Deep Reinforcement Learning

Deep Reinforcement Leaming for Short-term Voltage Control by Dynamic Load Shedding in China Southem Power Grid

Bridging Transient and Steady-State Performance in Voltage Control: A Reinforcement Learning Approach with Safe Gradient Flow

Stability Constrained Reinforcement Learning for Decentralized Real-Time Voltage Control

Combination optimization method of grid sections based on deep reinforcement learning with accelerated convergence speed

Deep Reinforcement Learning for Electric Transmission Voltage Control

Deep Reinforcement Learning for Optimizing Inverter Control: Fixed and Adaptive Gain Tuning Strategies for Power System Stability

An Intelligent Control Strategy for buck DC-DC Converter via Deep Reinforcement Learning

Robust Deep Reinforcement Learning for Volt-VAR Optimization in Active Distribution System under Uncertainty