Abstract:This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results for model-free DRL-based methods in power systems control problems. But in power systems applications, these model-free methods have certain issues related to training time (clock time) and sample efficiency; both are critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. It is also desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, the state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based DRL framework where a deep neural network (DNN)-based dynamic surrogate model (SM), instead of a real-world power grid or physics-based simulation, is utilized within the policy learning framework, making the process faster and more sample efficient. However, having stable training in model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We addressed these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step loss in surrogate model training. Finally, we achieved 97.5% reduction in samples and 87.7% reduction in training time for an application to the IEEE 300-bus test system.

Deep-Reinforcement-Learning-Based Autonomous Voltage Control for Power Grid Operations

Autonomous Voltage Control for Grid Operation Using Deep Reinforcement Learning

A Data-Driven Multi-Agent Autonomous Voltage Control Framework Using Deep Reinforcement Learning

Implementing Deep Reinforcement Learning-Based Grid Voltage Control in Real-World Power Systems: Challenges and Insights

Deep Reinforcement Learning for Electric Transmission Voltage Control

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Deep Reinforcement Learning Based Voltage Control Revisited

Coordination of PV Smart Inverters Using Deep Reinforcement Learning for Grid Voltage Regulation

Scalable Voltage Control using Structure-Driven Hierarchical Deep Reinforcement Learning

Deep Reinforcement Learning Based Approach for Active Power Security Correction Control of Power System

Efficient Learning of Voltage Control Strategies via Model-based Deep Reinforcement Learning

On Training Effective Reinforcement Learning Agents for Real-time Power Grid Operation and Control

A Date-Driven Voltage Control Strategy for Distribution Network Using Deep Reinforcement Learning

Deep-Reinforcement-Learning-Based Two-Timescale Voltage Control for Distribution Systems

A Deep Reinforcement Learning Voltage Control Method for Distribution Network

Deep Reinforcement Learning Based Robust Control Strategy for Power Sharing and Voltage Regulation in DC Grid-Forming Energy Storage System

Learning and Fast Adaptation for Grid Emergency Control via Deep Meta Reinforcement Learning

Graph Attention Network Based Deep Reinforcement Learning for Voltage/var Control of Topologically Variable Power System

Adaptive Power System Emergency Control using Deep Reinforcement Learning

Distributed Energy Management and Demand Response in Smart Grids: A Multi-Agent Deep Reinforcement Learning Framework

Data-Driven Multi-Agent Deep Reinforcement Learning for Distribution System Decentralized Voltage Control with High Penetration of PVs