Abstract:This paper develops a model-free volt-VAR optimization (VVO) algorithm via multi-agent deep reinforcement learning (MADRL) in unbalanced distribution systems. This method is novel since we cast the VVO problem in unbalanced distribution networks to an intelligent deep Q-network (DQN) framework, which avoids solving a specific optimization model directly when facing time-varying operating conditions of the systems. We consider statuses/ratios of switchable capacitors, voltage regulators, and smart inverters installed at distributed generators as the action variables of the DQN agents. A delicately designed reward function guides these agents to interact with the distribution system, in the direction of reinforcing voltage regulation and power loss reduction simultaneously. The forward-backward sweep method for radial three-phase distribution systems provides accurate power flow results within a few iterations to the DQN environment. Finally, the proposed multi-objective MADRL method realizes the dual goals for VVO. We test this algorithm on the unbalanced IEEE 13-bus and 123-bus systems. Numerical simulations validate the excellent performance of this method in voltage regulation and power loss reduction.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve reactive - voltage optimization (Volt - VAR Optimization, VVO) in unbalanced distribution systems, in order to improve the operating efficiency of the system and reduce power losses. Specifically, the paper proposes a model - independent VVO algorithm based on multi - agent deep reinforcement learning (Multi - Agent Deep Reinforcement Learning, MADRL). This algorithm can avoid directly solving specific optimization models in the face of system operating conditions that change over time, thereby achieving the dual goals of voltage regulation and power loss reduction. ### Main Problems and Challenges 1. **Limitations of Traditional Methods**: - Traditional VVO methods usually need to establish complex mixed - integer nonlinear programming (MINLP) models, which are non - convex and NP - hard. - The process of iteratively solving these models is time - consuming, especially in large - scale systems. - Centralized methods have high modeling complexity when dealing with three - phase unbalanced distribution systems and are difficult to adapt to real - time changing load conditions. 2. **Deficiencies of Existing Reinforcement Learning Methods**: - Existing methods based on single - agent reinforcement learning mainly focus on voltage regulation and ignore the optimization of power losses. - Single - agent methods have a slow learning speed and a huge search space when dealing with large - scale systems. ### Solutions in the Paper The paper proposes a VVO algorithm based on multi - agent deep Q - network (DQN), and the main contributions include: 1. **Multi - agent Framework**: - Take the states/proportions of various regulating devices (such as switched capacitors, voltage regulators and smart inverters) as the action variables of DQN agents. - Guide these agents to interact with the distribution system through a customized reward function to achieve the dual goals of voltage regulation and power loss reduction. 2. **Efficient Forward - Backward Scanning Method**: - Use the forward - backward scanning method to calculate the power flow of three - phase unbalanced distribution systems and provide accurate power flow results. - Integrate the ZIP load model and the models of regulating devices into the power flow calculation to improve the calculation efficiency. 3. **Model - Independence**: - This method does not need to establish a specific optimization model and can quickly adapt to changes in system operating conditions. ### Experimental Verification The paper was tested on the IEEE 13 - bus and 123 - bus unbalanced distribution systems, verifying the excellent performance of the proposed method in terms of voltage regulation and power loss reduction. The experimental results show that this method gradually optimizes the control strategy during the training process and can effectively deal with new operating conditions in the test stage. ### Conclusion The VVO algorithm based on multi - agent deep reinforcement learning proposed in this paper can achieve the dual goals of voltage regulation and power loss reduction in unbalanced distribution systems, and has high computational efficiency and adaptability.

Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems

Research on Volt/Var Control of Distribution Networks Based on PPO Algorithm

Robust Deep Reinforcement Learning for Volt-VAR Optimization in Active Distribution System under Uncertainty

Volt-VAR Control in Active Distribution Networks Using Multi-Agent Reinforcement Learning

Robust Regional Coordination of Inverter-Based Volt/Var Control Via Multi-Agent Deep Reinforcement Learning

Multi-agent Graph Reinforcement Learning for Decentralized Volt-VAR Control in Power Distribution Systems

Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks

Distributed Voltage Regulation of Active Distribution System Based on Enhanced Multi-agent Deep Reinforcement Learning

Deep Reinforcement Learning Based Coordinated Voltage Control in Smart Distribution Network

Two-Stage Deep Reinforcement Learning for Inverter-Based Volt-VAR Control in Active Distribution Networks

Residual Deep Reinforcement Learning with Model-based Optimization for Inverter-based Volt-Var Control

Research on the multi-timescale optimal voltage control method for distribution network based on a DQN-DDPG algorithm

Data-Driven Robust Voltage/VAR Control Using PV Inverters in Active Distribution Networks

Augmented Lagrangian-Based Safe Reinforcement Learning Approach for Distribution System Volt/VAR Control

Deep-Reinforcement-Learning-Based Two-Timescale Voltage Control for Distribution Systems

Multi-timescale voltage control for distribution system based on multi-agent deep reinforcement learning

Robust Offline Deep Reinforcement Learning for Volt-Var Control in Active Distribution Networks

Reinforcement Learning for Volt-Var Control: A Novel Two-stage Progressive Training Strategy

Multi-Agent Deep Reinforcement Learning for Voltage Control with Coordinated Active and Reactive Power Optimization

Data-Driven Volt/VAR Optimization for Modern Distribution Networks: A Review

Online Multi-Agent Reinforcement Learning for Decentralized Inverter-Based Volt-VAR Control