Many-Objective Distribution Network Reconfiguration Using Deep Reinforcement Learning-Assisted Optimization Algorithm

Yuanzheng Li,Yong Zhao,Lei Wu,Zhigang Zeng

2023-01-01

Abstract:In recent years, energy and environmental crises have been significant obstacles to the sustainable development of our society. In order to lessen these crises, renewable energy (RE) has been paid more attention to and utilized widely worldwide.

What problem does this paper attempt to address?

Many-Objective Distribution Network Reconfiguration via Deep Reinforcement Learning Assisted Optimization Algorithm

Yuanzheng Li,Guokai Hao,Yun Liu,Yaowen Yu,Zhixian Ni,Yong Zhao

DOI: https://doi.org/10.1109/tpwrd.2021.3107534

IF: 4.825

2022-06-01

IEEE Transactions on Power Delivery

Abstract:With the increasing penetration of renewable energy (RE), the operations of distribution network are threatened and some issues may appear, i.e., large voltage deviation, deterioration of statistic voltage stability, high power loss, etc. In turn, RE accommodation would be significantly impacted. Therefore, we propose a many-objective distribution network reconfiguration (MDNR) model, with the consideration of RE curtailment, voltage deviation, power loss, statistic voltage stability, and generation cost. This aims to assess the trade-off among these objectives for better operations of distribution networks. As the proposed model is a non-convex, non-linear, many-objective optimization problem, it is difficult to be solved. We further propose a deep reinforcement learning (DRL) assisted multi-objective bacterial foraging optimization (DRL-MBFO) algorithm. This algorithm combines the advantages of DRL and MBFO, and is targeted to find the Pareto front of proposed MDNR model with better searching efficiency. Finally, we conduct case study on the modified IEEE 33-bus, 69-bus, and 118-bus power distribution systems, and results verify the effectiveness of the MDNR model and outperformance of the proposed DRL-MBFO.

engineering, electrical & electronic
Multi-agent Deep Reinforcement Learning Algorithm for Distributed Economic Dispatch in Smart Grid.

Lifu Ding,Zhiyun Lin,Gangfeng Yan

DOI: https://doi.org/10.1109/iecon43393.2020.9255238

2020-01-01

Abstract:With the development of large-scale power grids, the issue of distributed economic dispatch has received considerable critical attention. However, due to the existence of some effects such as valve-point effects, the nonconvex objective function remains a major challenge for the distributed optimization problem. This paper proposes a cooperative deep reinforcement learning algorithm for distributed economic dispatch with the nonconvex objective function. In the distributed algorithm, all nodes obtain the value of actions by observing the environment and update state-action-value function in coordination with local neighbors. The state-action-value function is approximated by a neural network, which allows the algorithm to be used for large and continuous state spaces. The advantages of the algorithm are demonstrated through several case studies.
Target-Value-Competition-Based Multi-Agent Deep Reinforcement Learning Algorithm for Distributed Nonconvex Economic Dispatch

Lifu Ding,Zhiyun Lin,Xiasheng Shi,Gangfeng Yan

DOI: https://doi.org/10.1109/tpwrs.2022.3159825

IF: 7.326

2023-01-01

IEEE Transactions on Power Systems

Abstract:With the increasing expansion of the power grid, economic dispatch problems have received considerable attention. A multi-agent coordinated deep reinforcement learning algorithm is proposed to deal with distributed nonconvex economic dispatch problems. In the algorithm, agents run independent reinforcement learning algorithms and update their local Q-functions with a newly defined joint reward. The double network structure is adopted to approximate the Q-function so that the offline trained model can be used online to provide recommended power outputs for time-varying demands in real-time. By introducing the reward network, the competition mechanism between the reward network and the target network is established to determine a progressively stable target value, which achieves coordination among agents and pledges the losses of the Q-networks to converge well. Theoretical analysis is given and case studies are conducted to prove the advantages compared with existing approaches.
Multi-Agent Reinforcement Learning-based Distributed Economic Dispatch Considering Network attacks and Uncertain Costs

Dong Mao,Lifu Ding,Chen Zhang,Hanyu Rao,Gangfeng Yan

DOI: https://doi.org/10.1109/ICIEA51954.2021.9516143

2021-01-01

Abstract:With the development of large-scale power grids, distributed economic dispatch has received considerable attention. However, existing distributed economic dispatch algorithms ignore potential network attacks, which may have a greater impact on the security of the power grid. In addition, due to the access of renewable power generation equipment and external interference, there are usually uncertain terms in the cost function. A distributed economic dispatch collaborative deep reinforcement learning algorithm considering the safety objective function is proposed. All agents in the algorithm make joint decisions by observing the environment and coordinating with local neighbors. The state action value function is approximated by a neural network. Aiming at the problem caused by the uncertain term of the objective function, the idea of lenient reinforcement learning is adopted, and the reward is also fitted with a neural network. Several case studies have been conducted to prove the advantages of this algorithm.
A NEW MULTI-OBJECTIVE ALGORITHM FOR DISTRIBUTION NETWORK RECONFIGURATION

孙健,江道灼

DOI: https://doi.org/10.3321/j.issn:1000-1026.2003.20.013

2003-01-01

Abstract:Distribution network reconfiguration is an important content in distribution management system, which is a nonlinear combinatorial optimization problem in essence. It is difficult to obtain perfect results by traditional approach. A new algorithm for distribution network reconfiguration is proposed, which sufficiently utilizes and synthesizes load-balancing method and branch-exchanging method to improve load-balancing process and also uses a new partitioning power flow calculation method and a new heuristic rule deduced from the formula of branch-exchanging network loss estimation, and resolves the multi-objective distribution network reconfiguration problem properly. Theoretical analysis and numerical example show that the method is highly efficient and feasible, and conforms to the practical demands in distribution automation.
Reactive Power Optimization of Distribution Network Based on Deep Reinforcement Learning and Multi Agent System

Zhi Gao,Zhijie Zheng,Junyong Wu,Lujie Qi,Wensheng Li,Yang

DOI: https://doi.org/10.1109/ei252483.2021.9713395

2021-01-01

Abstract:This paper presents a reactive power optimization method of distribution network based on deep reinforcement learning (DRL) and multi-agent system. Firstly, a method of extracting distribution network fusion features is proposed, which extracts statistical features from the operation data of distribution network. Then, the statistical features and historical control strategy are taken as inputs, and the network loss and voltage deviation are taken as outputs respectively. The network loss agent and voltage deviation agent are trained. The reactive power optimization problem is transformed into a multi-step Markov decision-making process, and the Double DQN algorithm is used to construct DRL agent. A multi-agent system is composed of network loss and voltage deviation agents and DRL agents. Network loss and voltage deviation agents provide rewards for DRL agents, while DRL agents provide strategies for network loss and voltage deviation agents. They cooperate with each other to obtain the optimal strategy. The reactive power optimization control experiment of IEEE-37 bus distribution network is carried out. The results show that the proposed method can effectively reduce the node voltage offset and network loss. It has nothing to do with the model and parameters of the distribution network system, and has fast online decision-making speed. It can improve the operation economy of the distribution network.
DDPG-based Reactive Power Optimization Strategy for Active Distribution Network

Xinghua Liu,Xiaoyu Wang,Bangji Fan,Beibei Li,Ruilong Deng,Minru Kong

DOI: https://doi.org/10.1109/irce62232.2024.10739784

2024-01-01

Abstract:To tackle challenges such as voltage fluctuations and network losses arising from integrating distributed energy resources into the grid, this paper proposes a novel reactive power optimization approach for active distribution network based on deep reinforcement learning. Diverging from conventional methods, our approach models reactive power optimization as a constrained Markov decision process, encompassing the determination of reactive power outputs from photovoltaic sources and compensation devices. By employing a deep deterministic policy gradient algorithm, the agent learns to make daily decisions based on the state of the environment. The methodology incorporates distributed control by training the agent to formulate optimal coordination strategies for photovoltaic inverters and capacitors. The efficacy of our proposed method is demonstrated by a comprehensive analysis conducted on a modified IEEE 33-node distribution network. The results highlight the superior performance in reducing active losses and mitigating voltage offsets compared to conventional methods.
Optimal Operation Strategy for Distribution Grid with High Penetration Renewable Energy: a Novel Deep Reinforcement Learning Based Approach

Xianpeng Xi,Sichen Li,Di Cao,Zhenyuan Zhang,Qi Huang,Weihao Hu

DOI: https://doi.org/10.1109/psgec54663.2022.9880943

2022-01-01

Abstract:A high proportion of renewable energy sources like wind and photovoltaic are connected to the grid as an important measure to alleviate the global energy crisis. However, the intermittency and uncertainty of wind and PV increase the voltage fluctuation of the distribution grid and threaten the safety of its operation. To operate the distribution grid economically and reliably, a novel deep reinforcement learning algorithm is proposed in this paper to optimize the operation of distribution grid. First, an optimization model is created with the target of minimizing the operation cost and maintaining distribution grid reliable operation, taking into account the constraints of each node in the grid. The optimization model is then reformulated as a Markov decision process and solved using the soft actor-critic (SAC) algorithm. The proposed method in this paper has excellent adaptability in the face of multi-objective optimization problems and can also perform online real-time scheduling. A comparative evaluation of the proposed approach is conducted between several benchmark approaches utilizing real-world data in order to demonstrate its effectiveness.
Research on Distribution Network Reconfiguration Based on Deep Q-learning Network

Huifang Di,Yuqing Bao,Beibei Wang

DOI: https://doi.org/10.1109/ei250167.2020.9346939

2020-01-01

Abstract:The reconfiguration problem of distribution network is an optimization problem aims to minimize the transmission power loss. Traditional mathematical methods are difficult to solve this kind of nonlinear mixed integer problem. This paper proposes an algorithm based on deep Q-learning network (DQN) which combines deep Q-learning and artificial neural network. A new method for the distribution network reconfiguration is presented based on DQN in this paper, with the objective of minimizing transmission power loss and the consideration of the corresponding constraints. The proposed method has been proved valid by the results from the dynamic reconfiguration of the IEEE 33-bus test system.
Deep Q-Learning-based Distribution Network Reconfiguration for Reliability Improvement

Mukesh Gautam,Narayan Bhusal,Mohammed Benidris

2023-05-02

Abstract:Distribution network reconfiguration (DNR) has proved to be an economical and effective way to improve the reliability of distribution systems. As optimal network configuration depends on system operating states (e.g., loads at each node), existing analytical and population-based approaches need to repeat the entire analysis and computation to find the optimal network configuration with a change in system operating states. Contrary to this, if properly trained, deep reinforcement learning (DRL)-based DNR can determine optimal or near-optimal configuration quickly even with changes in system states. In this paper, a Deep Q Learning-based framework is proposed for the optimal DNR to improve reliability of the system. An optimization problem is formulated with an objective function that minimizes the average curtailed power. Constraints of the optimization problem are radial topology constraint and all nodes traversing constraint. The distribution network is modeled as a graph and the optimal network configuration is determined by searching for an optimal spanning tree. The optimal spanning tree is the spanning tree with the minimum value of the average curtailed power. The effectiveness of the proposed framework is demonstrated through several case studies on 33-node and 69-node distribution test systems.

Systems and Control
Distributed flexible resource regulation strategy for residential communities based on deep reinforcement learning

Tianyun Xu,Tao Chen,Ciwei Gao,Meng Song,Yishen Wang,Hao Yuan

DOI: https://doi.org/10.1049/gtd2.13284

2024-09-29

IET Generation Transmission & Distribution

Abstract:This paper proposes a deep reinforcement learning‐based distributed resource energy management strategy, taking into account the inherent physical and structural constraints of the distribution network. In an era characterized by the rapid proliferation of distributed flexible resources (DFRs), the development of customized energy management and regulation strategies has attracted significant interest from the field. The inherent geographical dispersion and unpredictability of these resources, however, pose substantial barriers to their effective and computationally tractable regulation. To address these impediments, this paper proposes a deep reinforcement learning‐based distributed resource energy management strategy, taking into account the inherent physical and structural constraints of the distribution network. This proposed strategy is modelled as a sequential decision‐making framework with a Markov decision process, informed by physical states and external information. In particular, targeting the community energy management system for critical public infrastructure and community holistic benefits maximization, the proposed approach proficiently adapts to fluctuations in resource variability and fluctuating market prices, ensuring intelligent regulation of distributed flexible resources. Simulation and empirical analysis demonstrate that the proposed deep reinforcement learning‐based strategy can improve the economic benefits and decision‐making efficiency of distributed flexible resource regulation while ensuring the security of distribution network power flow.

engineering, electrical & electronic
Deep reinforcement learning based multi-level dynamic reconfiguration for urban distribution network: A cloud-edge collaboration architecture

Siyuan Jiang,Hongjun Gao,Xiaohui Wang,Junyong Liu,Kunyu Zuo

DOI: https://doi.org/10.1016/j.gloei.2023.02.001

2023-03-08

Global Energy Interconnection

Abstract:With the construction of the power Internet of Things (IoT), communication between smart devices in urban distribution networks has been gradually moving towards high speed, high compatibility, and low latency, which provides reliable support for reconfiguration optimization in urban distribution networks. Thus, this study proposed a deep reinforcement learning based multi-level dynamic reconfiguration method for urban distribution networks in a cloud-edge collaboration architecture to obtain a real-time optimal multi-level dynamic reconfiguration solution. First, the multi-level dynamic reconfiguration method was discussed, which included feeder-, transformer-, and substation-levels. Subsequently, the multi-agent system was combined with the cloud-edge collaboration architecture to build a deep reinforcement learning model for multi-level dynamic reconfiguration in an urban distribution network. The cloud-edge collaboration architecture can effectively support the multi-agent system to conduct "centralized training and decentralized execution" operation modes and improve the learning efficiency of the model. Thereafter, for a multi-agent system, this study adopted a combination of offline and online learning to endow the model with the ability to realize automatic optimization and updation of the strategy. In the offline learning phase, a Q-learning-based multi-agent conservative Q-learning (MACQL) algorithm was proposed to stabilize the learning results and reduce the risk of the next online learning phase. In the online learning phase, a multi- agent deep deterministic policy gradient (MADDPG) algorithm based on policy gradients was proposed to explore the action space and update the experience pool. Finally, the effectiveness of the proposed method was verified through a simulation analysis of a real-world 445-node system.
Day-Ahead Optimal Dispatch in Active Distribution Network Based on Deep Reinforcement Learning with Improved Feature Extraction Network

Hanming Zhong,Peng Li,Jiahao Wang,Kecheng Li,Shaojie Zhu,Zhihao Yang

DOI: https://doi.org/10.1109/ei259745.2023.10512578

2023-01-01

Abstract:The access of renewable energy will challenge the economic and stable operation of active distribution network (AND). Based on the deep reinforcement learning algorithm, this paper proposes a multi-objective intelligent day-ahead optimal dispatch method for resources in distribution network. In the day-ahead optimal dispatch, the deep reinforcement learning (DRL) method is used to deal with the uncertainty of load and renewable resources. In order to achieve the economic operation of the distribution network and reduce the peak shaving pressure of the superior power grid, the regulation scheme of energy storage system and flexible load is formulated, and an improved feature extraction network is proposed to better deal with the problem of information redundancy. The effectiveness and superiority of this method are verified by an modified IEEE-33 example.
Optimal Scheduling of a Wind Energy Dominated Distribution Network Via a Deep Reinforcement Learning Approach

Jiaoyiling Zhu,Weihao Hu,Xiao Xu,Haoming Liu,Li Pan,Haoyang Fan,Zhenyuan Zhang,Zhe Chen

DOI: https://doi.org/10.1016/j.renene.2022.10.094

IF: 8.7

2022-01-01

Renewable Energy

Abstract:With the development of clean energy systems, large-scale renewable energy is being connected to the traditional distribution network, which also brings new challenges to the reliable and economic scheduling of the power grid. To address these challenges, this paper proposes an intelligent scheduling strategy for a wind energy dominated distribution network, which aims to reduce the fluctuation caused by the wind energy. First, the energy scheduling model and objective function of the distribution network system are established and the constraints of various types of components are considered. Then, deep reinforcement learning is introduced to realize real-time decision in distribution network to solve the problem of fluctuation caused by the uncertain wind power output. The energy scheduling method is developed into a Markov decision process based on deep deterministic policy gradient (DDPG) algorithm. Finally, the simulation is verified on the IEEE14 node system. The results verify that the proposed approach can effectively reduce power fluctuations in the distribution network. The superiority of the adopted DDPG algorithm is demonstrated by comparing with the deep Q network algorithm.
Overview On Algorithms Of Distribution Network Reconfiguration

Youjie Ma,Feng Liu,Xuesong Zhou,Zhiqiang Gao

DOI: https://doi.org/10.23919/chicc.2017.8029054

2017-01-01

Abstract:Reconfiguration of the distribution network is an important way to improve operation safety, economical efficiency and power supply quality, and it is of great significance to the construction and application of the distribution automation system ongoing in China. The problem of reconfiguration is a complicated combinatorial optimization problem with discrete decision variables. This paper introduces the optimization objective and constraints of the distribution network reconfiguration, and analyzes the complexity and difficulties in calculation, and overviews a variety of algorithms, including the heuristic, stochastic optimization and intelligent methods. In the end, the paper points out the research direction which deserves sufficient attention.
A Holistic Power Optimization Approach for Microgrid Control Based on Deep Reinforcement Learning

Fulong Yao,Wanqing Zhao,Matthew Forshaw,Yang Song

2024-11-02

Abstract:The global energy landscape is undergoing a transformation towards decarbonization, sustainability, and cost-efficiency. In this transition, microgrid systems integrated with renewable energy sources (RES) and energy storage systems (ESS) have emerged as a crucial component. However, optimizing the operational control of such an integrated energy system lacks a holistic view of multiple environmental, infrastructural and economic considerations, not to mention the need to factor in the uncertainties from both the supply and demand. This paper presents a holistic datadriven power optimization approach based on deep reinforcement learning (DRL) for microgrid control considering the multiple needs of decarbonization, sustainability and cost-efficiency. First, two data-driven control schemes, namely the prediction-based (PB) and prediction-free (PF) schemes, are devised to formulate the control problem within a Markov decision process (MDP). Second, a multivariate objective (reward) function is designed to account for the market profits, carbon emissions, peak load, and battery degradation of the microgrid system. Third, we develop a Double Dueling Deep Q Network (D3QN) architecture to optimize the power flows for real-time energy management and determine charging/discharging strategies of ESS. Finally, extensive simulations are conducted to demonstrate the effectiveness and superiority of the proposed approach through a comparative analysis. The results and analysis also suggest the respective circumstances for using the two control schemes in practical implementations with uncertainties.

Systems and Control
Research on Reactive Power Optimization Strategy under the Intelligent Improvement Model of the Distribution Network

Menglin Yu

DOI: https://doi.org/10.1155/2022/9310507

2022-10-04

Advances in Multimedia

Abstract:In order to improve the reactive power optimization effect of the distribution network, this paper combines the multiagent deep reinforcement learning algorithm to analyze the reactive power optimization strategy of the distribution network and constructs an intelligent optimization model. Moreover, the simulation models of power conversion elements, power transmission elements, control elements, and measurement elements in the platform are described, and the program structure and interactive functions are analyzed. In addition, this paper proposes a reactive power optimization method for distribution networks based on data-driven thinking. Finally, by using historical data and an artificial neural network, this paper extracts electrical quantity data such as load power and distributed power output and environmental data such as temperature and wind speed to perform multiagent analysis. The experimental verification shows that the reactive power optimization effect of the distribution network based on multiagent and multiagent deep reinforcement learning proposed in this paper is very good.
A Generalized Deep Reinforcement Learning Model for Distribution Network Reconfiguration with Power Flow-Based Action-Space Sampling

Nastaran Gholizadeh,Petr Musilek

DOI: https://doi.org/10.3390/en17205187

IF: 3.2

2024-10-19

Energies

Abstract:Distribution network reconfiguration (DNR) is used by utilities to enhance power system performance in various ways, such as reducing line losses. Conventional DNR algorithms rely on accurate values of network parameters and lack scalability and optimality. To tackle these issues, a new data-driven algorithm based on reinforcement learning is developed for DNR in this paper. The proposed algorithm comprises two main parts. The first part, named action-space sampling, aims at analyzing the network structure, finding all feasible reconfiguration actions, and reducing the size of the action space to only the most optimal actions. In the second part, deep Q-learning (DQN) and dueling DQN methods are used to train an agent to take the best switching actions according to the switch states and loads of the system. The results show that both DQN and dueling DQN are effective in reducing system losses through grid reconfiguration. The proposed methods have faster execution time compared to the conventional methods and are more scalable.

energy & fuels
Distribution Network Reconfiguration Based on NoisyNet Deep Q-Learning Network

Beibei Wang,Hong Zhu,Honghua Xu,Yuqing Bao,Huifang Di

DOI: https://doi.org/10.1109/access.2021.3089625

IF: 3.9

2021-01-01

IEEE Access

Abstract:The distribution network reconfiguration (DNR) aims at minimizing the power losses and improving the voltage profile. Traditional model-based methods exactly need the network parameters to derive the optimal configuration of the distribution network. This paper proposes a DNR method based on model-free reinforcement learning (RL) approach. The proposed method adopts NoisyNet deep Q-learning network (DQN), by which the exploration can be automatically realized without need of tuning the exploration parameters, in order to accelerate the training process and improve the optimization performance. The proposed method is validated by the simulation results.
AC/DC hybrid distribution network reconfiguration with microgrid formation using multi-agent soft actor-critic

Tao Wu,Jianhui Wang,Xiaonan Lu,Yuhua Du

DOI: https://doi.org/10.1016/j.apenergy.2021.118189

IF: 11.2

2022-02-01

Applied Energy

Abstract:Recent extreme events trigger tremendous concerns on distribution system resilience. Meanwhile, high penetration of inverter-interfaced distributed generators (DGs) and diversified source and load mix facilitate the development and implementation of hybrid AC and DC distribution networks (HDNs). This paper proposes a deep reinforcement learning-based (DRL) approach for distribution network reconfiguration with microgrid formation in face of extreme events. The proposed optimization model facilitates critical service restoration by forming isolated sections nested inside the HDNs when severe power outages occur (e.g., disconnection from the main grid). The operational characteristics of isolated HDNs (e.g., droop-controlled nodes in AC and DC sections, lack of slack buses in autonomous operation, etc.) are considered. To reduce the computational burden, a multi-agent soft actor-critic (MA-SAC) approach is developed to solve the proposed reconfiguration problem, where multiple agents coordinately control circuit breakers to sectionalize the HDNs and can cater for different system states and scales. Simulation tests are conducted in two test systems to verify the validity of the proposed approach.

energy & fuels,engineering, chemical

Many-Objective Distribution Network Reconfiguration Using Deep Reinforcement Learning-Assisted Optimization Algorithm

Many-Objective Distribution Network Reconfiguration via Deep Reinforcement Learning Assisted Optimization Algorithm

Multi-agent Deep Reinforcement Learning Algorithm for Distributed Economic Dispatch in Smart Grid.

Target-Value-Competition-Based Multi-Agent Deep Reinforcement Learning Algorithm for Distributed Nonconvex Economic Dispatch

Multi-Agent Reinforcement Learning-based Distributed Economic Dispatch Considering Network attacks and Uncertain Costs

A NEW MULTI-OBJECTIVE ALGORITHM FOR DISTRIBUTION NETWORK RECONFIGURATION

Reactive Power Optimization of Distribution Network Based on Deep Reinforcement Learning and Multi Agent System

DDPG-based Reactive Power Optimization Strategy for Active Distribution Network

Optimal Operation Strategy for Distribution Grid with High Penetration Renewable Energy: a Novel Deep Reinforcement Learning Based Approach

Research on Distribution Network Reconfiguration Based on Deep Q-learning Network

Deep Q-Learning-based Distribution Network Reconfiguration for Reliability Improvement

Distributed flexible resource regulation strategy for residential communities based on deep reinforcement learning

Deep reinforcement learning based multi-level dynamic reconfiguration for urban distribution network: A cloud-edge collaboration architecture

Day-Ahead Optimal Dispatch in Active Distribution Network Based on Deep Reinforcement Learning with Improved Feature Extraction Network

Optimal Scheduling of a Wind Energy Dominated Distribution Network Via a Deep Reinforcement Learning Approach

Overview On Algorithms Of Distribution Network Reconfiguration

A Holistic Power Optimization Approach for Microgrid Control Based on Deep Reinforcement Learning

Research on Reactive Power Optimization Strategy under the Intelligent Improvement Model of the Distribution Network

A Generalized Deep Reinforcement Learning Model for Distribution Network Reconfiguration with Power Flow-Based Action-Space Sampling

Distribution Network Reconfiguration Based on NoisyNet Deep Q-Learning Network

AC/DC hybrid distribution network reconfiguration with microgrid formation using multi-agent soft actor-critic