Abstract:Real-time power network dispatching and control (PDC) presents unique challenges that traditional methods cannot effectively address due to the consideration of temporal dynamic factors. Reinforcement learning (RL) has been introduced and proven to be effective. However, given the vast space for solutions, there is an urgent need to enhance the performance of RL further. Analyzing the characteristics of the power network to improve the performance of RL holds significant potential. This article presents a comprehensive analysis of power network characteristics, including imbalanced state distribution and the imbalanced action operation frequency, along with their impact on applying RL in real-time PDC to guide the algorithm design. Guided by the analysis, the paper proposes the Balance Deep Q-Network (DQN) algorithm to mitigate the negative impact of the imbalanced problems on algorithm performance. A K-means-based predefined curriculum learning (KPCL) module is proposed to address the state imbalance issue. It allows for differentiated access to operation scenarios based on their hardness, ensuring a balanced exploration of operation scenarios while maintaining diversity during training. An Option DQN module is proposed to tackle the action imbalance problem, utilizing a branching network structure and hierarchical action selection to reduce the coupling between actions with different usage frequencies, thereby enhancing the accuracy of action value estimation. The effectiveness of the Balance DQN algorithm is demonstrated through extensive experiments conducted on the Grid2Op platform with 14-bus and 36-bus cases. The results indicate that Balance DQN can efficiently alleviate the negative impacts of imbalanced problems on RL for real-time PDC agents and achieve good performance in both cases.

Hierarchical Reinforcement Learning for Power Network Topology Control

Multi-Agent Reinforcement Learning for Power Grid Topology Optimization

PowRL: A Reinforcement Learning Framework for Robust Management of Power Networks

Progressive Decision-Making Framework for Power System Topology Control

Managing power grids through topology actions: A comparative study between advanced rule-based and reinforcement learning agents

Hierarchical Task Planning for Power Line Flow Regulation

Scalable Voltage Control using Structure-Driven Hierarchical Deep Reinforcement Learning

Multi-agent hierarchical reinforcement learning for energy management

Alleviating Imbalanced Problems of Reinforcement Learning when Applying in Real-Time Power Network Dispatching and Control

Distributed Hierarchical Deep Reinforcement Learning for Large-Scale Grid Emergency Control

A Graph-based Deep Reinforcement Learning Framework for Autonomous Power Dispatch on Power Systems with Changing Topologies

Reinforcement Learning for Resilient Power Grids

Reinforcement Learning for Electricity Network Operation

Reconfiguring Unbalanced Distribution Networks Using Reinforcement Learning over Graphs

HUGO -- Highlighting Unseen Grid Options: Combining Deep Reinforcement Learning with a Heuristic Target Topology Approach

A prior knowledge-embedded reinforcement learning method for real-time active power corrective control in complex power systems

Optimizing Industrial HVAC Systems with Hierarchical Reinforcement Learning

Reinforcement Learning Based Power Grid Day-Ahead Planning and AI-Assisted Control

Flexible Power Load Control Based on Hierarchical Multi-Agent Reinforcement Learning

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Deep Reinforcement Learning for Multi-Agent Power Control in Heterogeneous Networks