Abstract:Real-time power network dispatching and control (PDC) presents unique challenges that traditional methods cannot effectively address due to the consideration of temporal dynamic factors. Reinforcement learning (RL) has been introduced and proven to be effective. However, given the vast space for solutions, there is an urgent need to enhance the performance of RL further. Analyzing the characteristics of the power network to improve the performance of RL holds significant potential. This article presents a comprehensive analysis of power network characteristics, including imbalanced state distribution and the imbalanced action operation frequency, along with their impact on applying RL in real-time PDC to guide the algorithm design. Guided by the analysis, the paper proposes the Balance Deep Q-Network (DQN) algorithm to mitigate the negative impact of the imbalanced problems on algorithm performance. A K-means-based predefined curriculum learning (KPCL) module is proposed to address the state imbalance issue. It allows for differentiated access to operation scenarios based on their hardness, ensuring a balanced exploration of operation scenarios while maintaining diversity during training. An Option DQN module is proposed to tackle the action imbalance problem, utilizing a branching network structure and hierarchical action selection to reduce the coupling between actions with different usage frequencies, thereby enhancing the accuracy of action value estimation. The effectiveness of the Balance DQN algorithm is demonstrated through extensive experiments conducted on the Grid2Op platform with 14-bus and 36-bus cases. The results indicate that Balance DQN can efficiently alleviate the negative impacts of imbalanced problems on RL for real-time PDC agents and achieve good performance in both cases.

A Generalized Deep Reinforcement Learning Model for Distribution Network Reconfiguration with Power Flow-Based Action-Space Sampling

Deep Q-Learning-based Distribution Network Reconfiguration for Reliability Improvement

Multi-agent Deep Reinforcement Learning Algorithm for Distributed Economic Dispatch in Smart Grid.

Distribution Network Reconfiguration Based on NoisyNet Deep Q-Learning Network

Explainable reinforcement learning for distribution network reconfiguration

Research on Distribution Network Reconfiguration Based on Deep Q-learning Network

A Multi-Agent Reinforcement Learning Method for Distribution System Restoration Considering Dynamic Network Reconfiguration

Sequential Reconfiguration of Unbalanced Distribution Network with Soft Open Points Based on Deep Reinforcement Learning

Safe Reinforcement Learning for Active Distribution Networks Reconfiguration Considering Uncertainty

DDPG-based Reactive Power Optimization Strategy for Active Distribution Network

DNOperNet: A Generic Network Architecture for Scalable and Safe Deep Reinforcement Learning in Distribution Network Operation

Many-Objective Distribution Network Reconfiguration via Deep Reinforcement Learning Assisted Optimization Algorithm

A Two-stage Multi-agent Deep Reinforcement Learning Method for Urban Distribution Network Reconfiguration Considering Switch Contribution

Reactive Power Optimization of Distribution Network Based on Deep Reinforcement Learning and Multi Agent System

Real-time autonomous dynamic reconfiguration based on deep learning algorithm for distribution network

Deep Reinforcement Learning Based Approach for Optimal Power Flow of Distribution Networks Embedded with Renewable Energy and Storage Devices

Deep Reinforcement Learning Based Approach for Dynamic Optimal Power Flow in Active Distribution Network

Alleviating Imbalanced Problems of Reinforcement Learning when Applying in Real-Time Power Network Dispatching and Control

Reconfiguring Unbalanced Distribution Networks Using Reinforcement Learning over Graphs

Real-Time Optimal Scheduling for Active Distribution Networks: A Graph Reinforcement Learning Method

Power Flow Rebalancing in Optimal Scheduling of Smart Distribution Systems Based on Deep Deterministic Policy Gradient