Abstract:Inverter-based Volt-VAR control (IB-VVC) can be simplified as a single-period optimization problem. However, recently DRL methods formulate IB-VVC as a Markov decision process (MDP) and solve it as a multi-period optimization problem. It complicates the IB-VVC problem and degrades the performance of deep reinforcement learning (DRL) algorithms. To avoid this, this paper formulates the inverter-based VVC as a one-step MDP and designs a single-period DRL algorithm to solve the problem. It simplifies the DRL approach considerably and accelerates the convergence rate as well as the control performance. Since VVC has two aims: eliminating voltage violations and minimizing power loss, those two objectives have different profiles. Existing DRL methods use one critic neural network to approximate the two objectives together without considering their special property. It increases the approximate difficulty of the critic neural networks in the training process. To alleviate it, we design a two-critic approach. It approximates the two objective functions by two critic neural networks separately. It has a better approximation capability, thus accelerating the convergence rate and improving the control performance of the DRL method further. Based on the single-period two-critic DRL (TC) approach, we design two DRL algorithms: 1) TC-DDPG with deterministic policy and 2) TC-SAC with stochastic policy. Further, we extend the TC-DRL to multi-agent TC to show it scales well for multi-agent DRL algorithms. Simulations conducted on 33-bus and 69-bus test distribution networks demonstrate the superiority of the proposed approach in both single-agent DRL algorithms and multi-agent DRL algorithms. A two-critic deep reinforcement learning (TC-DRL) approach for inverter-based volt-var control (IB-VVC) in active distribution networks is proposed in this paper. Considering two objectives of VVC, minimizing power loss and eliminating voltage violations, have different mathematical properties, we utilize two critics to approximate two objectives separately, which reduces the learning difficulties of each critic. The TC-DRL approach cooperates well with many actor-critic DRL algorithms for the centralized IB-VVC problems, and two centralized DRL algorithms were designed as examples. For decentralized IB-VVC, we extend the approach to a multi-agent TC-DRL approach and further simplify the multi-agent DRL approach with all agents sharing the same centralized two-critic. Extensive simulation experiments show that the proposed two centralized TC-DRL algorithms require fewer iteration times and return better results than the recent DRL algorithms, and the multi-agent TC-DRL algorithms work well for decentralized IB-VVC problems with different limited real-time measurement conditions.

A Novel Deep Reinforcement Learning-Based Current Control Method for Direct Matrix Converters

An Intelligent Control Strategy for buck DC-DC Converter via Deep Reinforcement Learning

Predictive Control of Voltage Source Inverter: an Online Reinforcement Learning Solution

A Model-Free Switching and Control Method for Three-Level Neutral Point Clamped Converter Using Deep Reinforcement Learning

AI-driven, Model-Free Current Control: A Deep Symbolic Approach for Optimal Induction Machine Performance

Data-Driven Finite Control-Set Model Predictive Control for Modular Multilevel Converter

Dynamic Control of Three-Phase Dual-Active-Bridge DC-DC Converters Based on Sliding-Mode-Learning Adaline

Predictor-Based Data-Driven Model-Free Adaptive Predictive Control of Power Converters Using Machine Learning

Learning-Based Model Predictive Control of DC-DC Buck Converters in DC Microgrids: A Multi-Agent Deep Reinforcement Learning Approach

Data-Enabled Finite State Predictive Control for Power Converters Via Adaline Neural Network

A Robust Deadbeat Predictive Current Control Method for IPMSM

Model-Free Moving-Discretized-Control-Set Predictive Control for Three-Phase Dual-Active-Bridge Converters

Deep Reinforcement Learning Based Control of a Grid Connected Inverter With LCL-Filter for Renewable Solar Applications

Deep Reinforcement Learning Based Input Voltage Sharing Method for Input-Series Output-Parallel Dual Active Bridge Converter in DC Microgrids

Two-Critic Deep Reinforcement Learning for Inverter-based Volt-Var Control in Active Distribution Networks

Reinforcement Learning Based Integrated Control to Improve the Efficiency of DC Microgrids

Reinforcement Learning-Based Control of a Power Electronic Converter

Reinforcement learning-based control of improved hybrid current modulated dual active bridge AC/DC converter

Meta-Reinforcement-Learning-Based Current Control of Permanent Magnet Synchronous Motor Drives for a Wide Range of Power Classes

Deep Reinforcement Learning Based Control Strategy for Voltage Regulation of DC-DC Buck Converter Feeding CPLs in DC Microgrid

Deep Reinforcement Learning for Optimizing Inverter Control: Fixed and Adaptive Gain Tuning Strategies for Power System Stability