Abstract: In Volt/Var control (VVC) of active distribution networks(ADNs), both slow timescale discrete devices (STDDs) and fast timescale continuous devices (FTCDs) are involved. The STDDs such as on-load tap changers (OLTC) and FTCDs such as distributed generators should be coordinated in time sequence. Such VCC is formulated as a two-timescale optimization problem to jointly optimize FTCDs and STDDs in ADNs. Traditional optimization methods are heavily based on accurate models of the system, but sometimes impractical because of their unaffordable effort on modelling. In this paper, a novel bi-level off-policy reinforcement learning (RL) algorithm is proposed to solve this problem in a model-free manner. A Bi-level Markov decision process (BMDP) is defined to describe the two-timescale VVC problem and separate agents are set up for the slow and fast timescale sub-problems. For the fast timescale sub-problem, we adopt an off-policy RL method soft actor-critic with high sample efficiency. For the slow one, we develop an off-policy multi-discrete soft actor-critic (MDSAC) algorithm to address the curse of dimensionality with various STDDs. To mitigate the non-stationary issue existing the two agents' learning processes, we propose a multi-timescale off-policy correction (MTOPC) method by adopting importance sampling technique. Comprehensive numerical studies not only demonstrate that the proposed method can achieve stable and satisfactory optimization of both STDDs and FTCDs without any model information, but also support that the proposed method outperforms existing two-timescale VVC methods.

Reinforcement Learning for Volt-Var Control: A Novel Two-stage Progressive Training Strategy

Research on Volt/Var Control of Distribution Networks Based on PPO Algorithm

Multi-agent Graph Reinforcement Learning for Decentralized Volt-VAR Control in Power Distribution Systems

Simplified Deep Reinforcement Learning Based Volt-var Control of Topologically Variable Power System

Two-Stage Deep Reinforcement Learning for Inverter-Based Volt-VAR Control in Active Distribution Networks

Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks

Two-Stage Deep Reinforcement Learning for Distribution System Voltage Regulation and Peak Demand Management

Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems

Three-Stage Inverter-Based Peak Shaving and Volt-VAR Control in Active Distribution Networks Using Online Safe Deep Reinforcement Learning

Volt-VAR Control in Active Distribution Networks Using Multi-Agent Reinforcement Learning

Bi-level Off-policy Reinforcement Learning for Volt/VAR Control Involving Continuous and Discrete Devices

Residual Deep Reinforcement Learning for Inverter-based Volt-Var Control

Robust Regional Coordination of Inverter-Based Volt/Var Control Via Multi-Agent Deep Reinforcement Learning

Coordination of PV Smart Inverters Using Deep Reinforcement Learning for Grid Voltage Regulation

Online Volt-Var Control Based on Reinforcement Learning in Active Distribution Systems Considering PV Inverters

Bi-level Off-policy Reinforcement Learning for Two-Timescale Volt/VAR Control in Active Distribution Networks

Data-Driven Robust Voltage/VAR Control Using PV Inverters in Active Distribution Networks

Residual Deep Reinforcement Learning with Model-based Optimization for Inverter-based Volt-Var Control

Volt/Var Control Method Based on Agents Group Using Reinforcement Learning

Robust Offline Deep Reinforcement Learning for Volt-Var Control in Active Distribution Networks

A Multi-Agent Deep Reinforcement Learning Based Voltage Regulation Using Coordinated PV Inverters