Abstract:Learning in high-dimensional action spaces is a key challenge in applying reinforcement learning (RL) to real-world systems. In this paper, we study the possibility of controlling power networks using RL methods. Power networks are critical infrastructures that are complex to control. In particular, the combinatorial nature of the action space poses a challenge to both conventional optimizers and learned controllers. Hierarchical reinforcement learning (HRL) represents one approach to address this challenge. More precisely, a HRL framework for power network topology control is proposed. The HRL framework consists of three levels of action abstraction. At the highest level, there is the overall long-term task of power network operation, namely, keeping the power grid state within security constraints at all times, which is decomposed into two temporally extended actions: 'do nothing' versus 'propose a topology change'. At the intermediate level, the action space consists of all controllable substations. Finally, at the lowest level, the action space consists of all configurations of the chosen substation. By employing this HRL framework, several hierarchical power network agents are trained for the IEEE 14-bus network. Whereas at the highest level a purely rule-based policy is still chosen for all agents in this study, at the intermediate level the policy is trained using different state-of-the-art RL algorithms. At the lowest level, either an RL algorithm or a greedy algorithm is used. The performance of the different 3-level agents is compared with standard baseline (RL or greedy) approaches. A key finding is that the 3-level agent that employs RL both at the intermediate and the lowest level outperforms all other agents on the most difficult task. Our code is publicly available.

Efficient Power Grid Topology Control via Two-Stage Action Search

Progressive Decision-Making Framework for Power System Topology Control

HUGO -- Highlighting Unseen Grid Options: Combining Deep Reinforcement Learning with a Heuristic Target Topology Approach

Managing power grids through topology actions: A comparative study between advanced rule-based and reinforcement learning agents

Imitation Learning for Intra-Day Power Grid Operation through Topology Actions

Hierarchical Reinforcement Learning for Power Network Topology Control

Multi-Agent Reinforcement Learning for Power Grid Topology Optimization

Action Set Based Policy Optimization for Safe Power Grid Management

State and Action Factorization in Power Grids

A Graph-based Deep Reinforcement Learning Framework for Autonomous Power Dispatch on Power Systems with Changing Topologies

On Training Effective Reinforcement Learning Agents for Real-time Power Grid Operation and Control

Network Topology Optimization via Deep Reinforcement Learning

Population-based Multi-agent Evaluation for Large-scale Voltage Control.

Controlling Directed Networks with Evolving Topologies

Power grid optimal topology dispatch and early fault warning using GAT and HDRL

Combination optimization method of grid sections based on deep reinforcement learning with accelerated convergence speed

Active power correction strategies based on deep reinforcement learning-part I: A simulation-driven solution for robustness

Power Grid Congestion Management via Topology Optimization with AlphaZero

Reinforcement Learning for Resilient Power Grids

Hierarchical Task Planning for Power Line Flow Regulation

A prior knowledge-embedded reinforcement learning method for real-time active power corrective control in complex power systems