Federated Multi-agent Deep Reinforcement Learning for Multi-microgrid Energy Management

Yuanzheng Li,Yong Zhao,Lei Wu,Zhigang Zeng

2023-01-01

Abstract:In recent years, renewable energy (RE) has been widely deployed, such as wind power and photovoltaic. Unlike traditional energy, RE resources are usually distributed.

What problem does this paper attempt to address?

Multi-agent Deep Reinforcement Learning Algorithm for Distributed Economic Dispatch in Smart Grid.

Lifu Ding,Zhiyun Lin,Gangfeng Yan

DOI: https://doi.org/10.1109/iecon43393.2020.9255238

2020-01-01

Abstract:With the development of large-scale power grids, the issue of distributed economic dispatch has received considerable critical attention. However, due to the existence of some effects such as valve-point effects, the nonconvex objective function remains a major challenge for the distributed optimization problem. This paper proposes a cooperative deep reinforcement learning algorithm for distributed economic dispatch with the nonconvex objective function. In the distributed algorithm, all nodes obtain the value of actions by observing the environment and update state-action-value function in coordination with local neighbors. The state-action-value function is approximated by a neural network, which allows the algorithm to be used for large and continuous state spaces. The advantages of the algorithm are demonstrated through several case studies.
Hybrid Multi-Agent Reinforcement Learning for Electric Vehicle Resilience Control Towards a Low-Carbon Transition

Dawei Qiu,Yi Wang,Tingqi Zhang,Mingyang Sun,Goran Strbac

DOI: https://doi.org/10.1109/tii.2022.3166215

IF: 12.3

2022-01-01

IEEE Transactions on Industrial Informatics

Abstract:In responseto low-carbon requirements, a large amount of renewable energy resources (RESs) have been deployed in power systems; nevertheless, the intermittency of RESs raises the system vulnerability and even causes severe damage under extreme events. Electric vehicles (EVs), owing to their mobility and flexibility characteristics, can provide various ancillary services meanwhile enhancing system resilience. The distributed control of EVs under such scenarios in power-transportation network becomes a complex decision-making problem with enormous dynamics and uncertainties. To this end, a multiagent reinforcement learning method is proposed to compute both discrete and continuous actions simultaneously that aligns with the nature of EV routing and scheduling problems. Furthermore, the proposed method can enhance the learning stability and scalability with privacy perseverance in the multiagent setting. Simulation results based on IEEE 6- and 33-bus power networks integrated with transportation systems validate its effectiveness in providing system resilience and carbon intensity service.
Multi-energy Management of Interconnected Multi-microgrid System Using Multi-agent Deep Reinforcement Learning

Sichen Li,Di Cao,Weihao Hu,Qi Huang,Zhe Chen,Frede Blaabjerg

DOI: https://doi.org/10.35833/MPCE.2022.000473

IF: 4.469

2023-01-01

Journal of Modern Power Systems and Clean Energy

Abstract:The multi-directional flow of energy in a multi-microgrid (MMG) system and different dispatching needs of multiple energy sources in time and location hinder the optimal operation coordination between microgrids. We propose an approach to centrally train all the agents to achieve coordinated control through an individual attention mechanism with a deep dense neural network for reinforcement learning. The attention mechanism and novel deep dense neural network allow each agent to attend to the specific information that is most relevant to its reward. When training is complete, the proposed approach can construct decisions to manage multiple energy sources within the MMG system in a fully decentralized manner. Using only local information, the proposed approach can coordinate multiple internal energy allocations within individual microgrids and external multilateral multi-energy interactions among interconnected microgrids to enhance the operational economy and voltage stability. Comparative results demonstrate that the cost achieved by the proposed approach is at most 71.1% lower than that obtained by other multi-agent deep reinforcement learning approaches.
Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning

Daniel J.B. Harrold,Jun Cao,Zhong Fan

DOI: https://doi.org/10.1016/j.apenergy.2022.119151

IF: 11.2

2022-07-15

Applied Energy

Abstract:To reduce global greenhouse gas emissions, the world must find intelligent solutions to maximise the utilisation of carbon-free renewable energy sources. In this paper, multi-agent reinforcement learning is used to control a microgrid in a mixed cooperative and competitive setting. The agents observe fluctuating energy demand, dynamic wholesale energy prices, and intermittent renewable energy sources to control a hybrid energy storage system to maximise the utilisation of the renewables to reduce the energy costs of the grid. In addition, an aggregator agent trades with external microgrids competing against one another and the aggregator to reduce their own energy bills. For this, the algorithm deep deterministic policy gradients (DDPG) and multi-agent DDPG (MADDPG) are used to compare the use of a single global controller versus multiple distributed agents, along with the single and multi-agent variants of distributional DDPG (D3PG) and twin delayed DDPG (TD3). The research found it is significantly more profitable for the primary microgrid to sell energy on its own terms rather than selling back to the utility grid, and is also beneficial for the external microgrids as they also reduce their own energy bills. The methods that produced the greatest profits were the multi-agent approaches where each agent has its own reward function based on the principle of marginal contribution from game theory. The multi-agent approaches were better able to evaluate their performance controlling individual components of the environment which allowed them to develop their own unique policies for the different types of energy storage system.

energy & fuels,engineering, chemical
Multi-agent Deep Reinforcement Learning for Distributed Energy Management and Strategy Optimization of Microgrid Market

Xiaohan Fang,Qiang Zhao,Jinkuan Wang,Yinghua Han,Yuchun Li

DOI: https://doi.org/10.1016/j.scs.2021.103163

IF: 11.7

2021-11-01

Sustainable Cities and Society

Abstract:<p>With the increasing demands for autonomous decision, market initiative, and privacy protection, a distributed energy management and strategy-making framework is more appropriate for the future microgrid. Besides, the integration of various distributed generations (DGs) and load facilities with high randomness poses challenges to traditional model-free microgrid market management approaches. Therefore, this paper concentrates on the distributed energy management and strategy optimization for a regional microgrid from the following aspects. First, a multi-agent reinforcement learning (MARL) framework is applied to generate independent real-time market decisions and to keep the benefits balance during agents' interactive learning. Moreover, to solve the defects of traditional RL in the microgrid case of continuous state inputs, a multi-agent deep Q-network (MADQN) is adopted as the approximation of value function. Finally, an optimal equilibrium selection (OES) mechanism is proposed to calculate the collective update objective for promoting learning efficiency and to ensure benefits equilibrium. Combining theoretical analysis and simulation results, the proposed MARL framework can improve the microgrid operation performance from economy, independence, and benefit balance among microgrid participants.</p>

energy & fuels,construction & building technology,green & sustainable science & technology
Federated Multiagent Deep Reinforcement Learning Approach via Physics-Informed Reward for Multimicrogrid Energy Management

Yuanzheng Li,Shangyang He,Yang Li,Yang Shi,Zhigang Zeng

DOI: https://doi.org/10.1109/tnnls.2022.3232630

IF: 14.255

2023-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:The utilization of large-scale distributed renewable energy (RE) promotes the development of the multimicrogrid (MMG), which raises the need of developing an effective energy management method to minimize economic costs and keep self energy sufficiency. The multiagent deep reinforcement learning (MADRL) has been widely used for the energy management problem because of its real-time scheduling ability. However, its training requires massive energy operation data of microgrids (MGs), while gathering these data from different MGs would threaten their privacy and data security. Therefore, this article tackles this practical yet challenging issue by proposing a federated MADRL (F-MADRL) algorithm via the physics-informed reward. In this algorithm, the federated learning (FL) mechanism is introduced to train the F-MADRL algorithm, thus ensures the privacy and the security of data. In addition, a decentralized MMG model is built, and the energy of each participated MG is managed by an agent, which aims to minimize economic costs and keep self energy sufficiency according to the physics-informed reward. At first, MGs individually execute the self-training based on local energy operation data to train their local agent models. Then, these local models are periodically uploaded to a server and their parameters are aggregated to build a global agent, which will be broadcasted to MGs and replace their local agents. In this way, the experience of each MG agent can be shared and the energy operation data are not explicitly transmitted, thus protecting the privacy and ensuring data security. Finally, experiments are conducted on Oak Ridge National Laboratory distributed energy control communication laboratory MG (ORNL-MG) test system, and the comparisons are carried out to verify the effectiveness of introducing the FL mechanism and the outperformance of our proposed F-MADRL.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Federated Multi-Agent Deep Reinforcement Learning Approach via Physics-Informed Reward for Multi-Microgrid Energy Management

Yuanzheng Li,Shangyang He,Yang Li,Yang Shi,Zhigang Zeng

DOI: https://doi.org/10.1109/TNNLS.2022.3232630

2022-12-29

Abstract:The utilization of large-scale distributed renewable energy promotes the development of the multi-microgrid (MMG), which raises the need of developing an effective energy management method to minimize economic costs and keep self energy-sufficiency. The multi-agent deep reinforcement learning (MADRL) has been widely used for the energy management problem because of its real-time scheduling ability. However, its training requires massive energy operation data of microgrids (MGs), while gathering these data from different MGs would threaten their privacy and data security. Therefore, this paper tackles this practical yet challenging issue by proposing a federated multi-agent deep reinforcement learning (F-MADRL) algorithm via the physics-informed reward. In this algorithm, the federated learning (FL) mechanism is introduced to train the F-MADRL algorithm thus ensures the privacy and the security of data. In addition, a decentralized MMG model is built, and the energy of each participated MG is managed by an agent, which aims to minimize economic costs and keep self energy-sufficiency according to the physics-informed reward. At first, MGs individually execute the self-training based on local energy operation data to train their local agent models. Then, these local models are periodically uploaded to a server and their parameters are aggregated to build a global agent, which will be broadcasted to MGs and replace their local agents. In this way, the experience of each MG agent can be shared and the energy operation data is not explicitly transmitted, thus protecting the privacy and ensuring data security. Finally, experiments are conducted on Oak Ridge national laboratory distributed energy control communication lab microgrid (ORNL-MG) test system, and the comparisons are carried out to verify the effectiveness of introducing the FL mechanism and the outperformance of our proposed F-MADRL.

Systems and Control,Machine Learning
Multi-agent deep reinforcement learning based distributed control architecture for interconnected multi-energy microgrid energy management and optimization

Bin Zhang,Weihao Hu,Amer M.Y.M. Ghias,Xiao Xu,Zhe Chen

DOI: https://doi.org/10.1016/j.enconman.2022.116647

IF: 10.4

2023-02-01

Energy Conversion and Management

Abstract:Environmental and climate change concerns are pushing the rapid development of new energy resources (DERs). The Energy Internet (EI), with the power-sharing functionality introduced by energy routers (ERs), offers an appealing alternative for DER systems. However, previous centralized control schemes for EI systems that follow a top-down architecture are unreliable for future power systems. This study proposes a distributed control scheme for bottom-up EI architecture. Second, model-based distributed control methods are not sufficiently flexible to deal with the complex uncertainties associated with multi-energy demands and DERs. A novel model-free/data-driven multiagent deep reinforcement learning (MADRL) method is proposed to learn the optimal operation strategy for the bottom-layer microgrid (MG) cluster. Unlike existing single-agent deep reinforcement learning methods that rely on homogeneous MG settings, the proposed MADRL adopts a form of decentralized execution, in which agents operate independently to meet local customized energy demands while preserving privacy. Third, an attention mechanism is added to the centralized critic, which can effectively accelerate the learning speed. Considering the bottom-layer power exchange request and the predicted electricity price, model predictive control of the upper layer determines the optimal power dispatching between the ERs and main grid. Simulations with other alternatives demonstrate the effectiveness of the proposed control scheme.

energy & fuels,thermodynamics,mechanics
Multi-agent Deep Reinforcement Learning Strategy for Distributed Energy

Lei Xi,Mengmeng Sun,Huan Zhou,Yanchun Xu,Junnan Wu,Yanying Li

DOI: https://doi.org/10.1016/j.measurement.2021.109955

IF: 5.6

2021-01-01

Measurement

Abstract:The strong random disturbance issues caused by the large-scale grid connections of distributed energy, such as wind energy, photovoltaic energy storage and electric vehicles, must be resolved. In this paper, we propose a Multi-agent deep reinforcement learning strategy, namely DDQN-CDP, which deeply integrate the improved actor-critic strategy with the neural network. This approach also solves the problem of the lack of continuous action controlling ability of traditional deep reinforcement learning, and obtains an optimal solution by multiregion collaboration. By simulating the modified IEEE standard two-area load frequency control power system model and Hubei power grid model, our results indicate that the proposed strategy can solve the strong random disturbance problem caused by the large-scale grid connections of distributed energy and achieve faster convergence and better control performance than other strategies.
A Multi-Agent Deep Reinforcement Learning Approach Enabled Distributed Energy Management Schedule for the Coordinate Control of Multi-Energy Hub with Gas, Electricity, and Freshwater

Guozhou Zhang,Weihao Hu,Di Cao,Zhenyuan Zhang,Qi Huang,Zhe Chen,Frede Blaabjerg

DOI: https://doi.org/10.1016/j.enconman.2022.115340

IF: 10.4

2022-01-01

Energy Conversion and Management

Abstract:In recent years, due to the deeply concerns on environment protection, the production, transformation and utilization of the energy sources with a more efficient and various way has become an important research topic. Under this background, this paper designs a renewable energy powered multi-energy hub system with gas, electricity, and freshwater sub-system. To enhance the flexibility and reduce the total cost of such system, the energy management of the multi-energy hub is formed as multi-agent cooperative control, and several targets, including operational costs, environment cost, and maintenance cost are considered along with the constraints. Subsequently, a novel attention mechanism-based multi-agent deep reinforcement learning algorithm is applied, where multi-agents are centrally trained to obtain the coordinate energy management strategy while being executed in a decentralized manner to provide the dispatch instruction for each energy hub with only local states. Finally, the effectiveness of the proposed method is investigated in the study system and the simulation results show that it can reduce the total cost by up to 7.28% compared with the other benchmark methods.
Distributed Energy Trading and Scheduling Among Microgrids via Multiagent Reinforcement Learning

Guanyu Gao,Yonggang Wen,Dacheng Tao

DOI: https://doi.org/10.1109/tnnls.2022.3170070

IF: 14.255

2022-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Renewable energy technologies empower microgrids to generate electricity to supply themselves and trade with others. Under this paradigm, microgrids have become autonomous entities that must intelligently determine their policies for energy trading and scheduling. Many factors influence a microgrid's decision-making, such as the complex microgrid infrastructure, the uncertain energy yield and demand, and the competition among the energy market players. These factors are usually hard to precisely model, and deriving the optimal policy for a microgrid is challenging. We propose a multiagent reinforcement learning (MARL) approach with an attention mechanism to learn the optimal policies for the microgrids without complex system modeling. We model each microgrid as an autonomous agent, which learns how to schedule energy resources and trade with others by collaborating with other agents. We adopt attention mechanism to enable intelligently selecting contextual information for the training of each agent. After training, an agent can make control decisions using only its local information, which can well preserve the microgrids' privacy and reduce the communication overhead among microgrids to facilitate distributed control. We implement a simulation environment and evaluate the performances of our proposed method using real-world datasets. The experimental results show that our method can significantly reduce the cost of the microgrids compared with the baseline methods.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Federated Deep Q-Network for Multiple Microgrids Energy Management

Bin Feng,Gang Huang,Yuanxi Chen,Zhitong Guo,Wenhao Zhu,Chuangxin Guo,Qiang Yang

DOI: https://doi.org/10.1109/icpst61417.2024.10602221

2024-01-01

Abstract:The unpredictability of renewable energy sources and load fluctuations pose significant challenges to effective microgrid energy management. Current model-based MG energy management methods are inefficient as they fail to fully utilize massive data and model the system. In contrast, deep reinforcement learning (DRL) can derive effective energy management strategies from historical data and dynamically adapt to various scenarios. However, existing DRL methods pose a risk of potential exposure of sensitive information due to the processing of large amounts of user data. In this paper, we propose the federated deep Q-network approach to address the potential data privacy breaches during DRL training and enhance energy management strategies. Firstly, the MG energy management problem is formulated as a Markov decision process, aiming to maximize the profits of multiple MG operators. Secondly, we introduce the federated deep Q-network approach, which combines the advantages of federated learning and the DQN algorithm. Case studies demonstrate that the proposed federated deep Q-network method is effective and rapidly convergent during training. This method only requires local gradient data from each MG, thereby ensuring the privacy of each MG user.
Distributed energy management of multi-area integrated energy system based on multi-agent deep reinforcement learning

Lifu Ding,Youkai Cui,Gangfeng Yan,Yaojia Huang,Zhen Fan

DOI: https://doi.org/10.1016/j.ijepes.2024.109867

IF: 5.659

2024-06-01

International Journal of Electrical Power & Energy Systems

Abstract:This paper addresses the problem of distributed energy management in multi-area integrated energy systems (MA-IES) using a multi-agent deep reinforcement learning approach. The MA-IES consists of interconnected electric and thermal networks, incorporating renewable energy sources and heat conversion systems. The objective is to optimize the operation of the system while minimizing operational costs and maximizing renewable energy utilization. We propose a distributed energy management strategy that makes hierarchical decisions on intra-area heat energy and inter-area electric energy. The strategy is based on a multi-agent deep reinforcement learning framework, where each agent represents a component or unit in the MA-IES. We formulate the problem as a Markov Decision Process and employ Q-learning with experience replay and double networks to train the agents. The proposed strategy is evaluated using a simulation of a four-area MA-IES. The results demonstrate significant improvements in energy management compared to traditional methods, with higher renewable energy utilization and lower operational costs. Specifically, the strategy achieves 100% utilization of wind power, and decreases operational costs by 5.53%. Furthermore, it leverages the generalization capabilities of reinforcement learning to respond in real-time to uncertainties in demand and wind power output. The results highlight the advantages of the proposed strategy, making it a promising solution for optimizing the operation of multi-area integrated energy systems.

engineering, electrical & electronic
Multi-agent deep reinforcement learning-based optimal energy management for grid-connected multiple energy carrier microgrids

Farzam Monfaredi,Hossein Shayeghi,Pierluigi Siano

DOI: https://doi.org/10.1016/j.ijepes.2023.109292

2023-06-13

Abstract:Multi-agent deep reinforcement learning (MA-DRL) method provides a groundbreaking approach to tackling computational problems in power systems, particularly for distributed energy resources that have been widely adopted to advance energy sustainability. This paper presents a novel optimal energy management based on proposed MA-DRL method. This method employs deep neural network to learn strategy based on stacked-denoising auto-encoders and multi-agent deep deterministic policy gradient learning capability. The MA-DRL method is adopted to find the optimal strategy of the optimal energy management problem under the Markov decision process framework. This method aims to coordinate multiple energies and achieve optimal operation over a variety of hourly dispatches while taking into account the distinct properties of electric and thermal energies. The primary challenge of the planning and operation of multiple energy carrier microgrids (MECMs) is determining the optimal interaction between renewable energy resources, energy storage systems, power-to-thermal conversion systems, and upstream power grid in order to improve overall energy utilization efficiency. The presented robust method can adaptively derive the optimal operation for MECMs through centralized learning and decentralized implementation. The optimization problem is employed in this study to concurrently reduce the total emissions and the operating costs while considering engineering design constraints. Finally, to demonstrate the efficiency of the proposed method, it is verified on an integrated modified IEEE 33-bus and 8-node gas systems.

engineering, electrical & electronic
Energy Optimization Management of Multi-microgrid Using Deep Reinforcement Learning

Tingjun Zhang,Dong Yue,Nan Zhao

DOI: https://doi.org/10.1109/cac51589.2020.9326507

2020-01-01

Abstract:The paper investigates the energy optimization management problem for multi-microgrid (MMG) in the island case. MMG system could achieve greater consumption of distributed energy and a more stable power supply. While, due to the existence of uncertainties such as wind turbines, photovoltaics and loads, it is challenging to design an accurate energy optimization model to control power flow. Notably, conventional methods also appear to be inapplicable. To address the problem of MMG, a deep reinforcement learning optimal algorithm is applied in this paper. The simulation verifies that the proposed method could achieve the power balance within MMG and could minimize the total cost under uncertainties.
Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system

Yuan Gao,Yuki Matsunami,Shohei Miyata,Yasunori Akashi

DOI: https://doi.org/10.1016/j.apenergy.2022.120021

IF: 11.2

2022-01-01

Applied Energy

Abstract:With the application of renewable energy in building energy systems (BES), an increasing number of power grids require building energy systems coupled to realize off-grid operation which is one type of energy flexible and grid responsive operations. In this case, deep reinforcement learning (DRL) algorithms have gained more and more attention in the operation control of BES due to their strong fitting ability and model-free utilization characteristics. However, mainstream DRL algorithms cannot solve the reinforcement learning problem of hybrid action spaces, which also restricts its further application in BES including variety of energy sources. In this paper, we firstly use a multi-agent deep reinforcement learning algorithm (MADRL) to solve the RL problem of hybrid action spaces in the building controls domain. The proposed algorithm is validated on a measured dataset of a real office building in Japan. The results show that compared to the currently used baseline control logic, MADRL can achieve a 60% improvement in off-grid operation tasks. For battery safety, MADRL can reduce unsafe battery runtime by at least 80%. Furthermore, through experiments we find that the single-agent DRL algorithm cannot solve the reinforcement learning problem with hybrid action spaces. The MADRL framework achieves stable training and optimization by layering problems and agents.
Multiagent deep meta reinforcement learning for sea computing-based energy management of interconnected grids considering renewable energy sources in sustainable cities

Jiawen Li,Tao Zhou

DOI: https://doi.org/10.1016/j.scs.2023.104917

IF: 11.7

2023-09-10

Sustainable Cities and Society

Abstract:In a sustainable city with a large amount of renewable energy, the difficulty inherent in the coordination of the load frequency control strategies of grid units and area units lead to severe frequency fluctuations. Conventional load frequency control has difficulty overcoming the above problems due to its lack of adaptive control ability and robustness. To overcome this problem, this paper proposes a sea computing-based grid-area coordinated load frequency control (SCGAC-LFC) method, which equates each grid unit and area unit in each area as independent agents. Instead of different units relying on different strategies, all units collectively obtain the LFC policy that suits the market requirements. In its online application, no communication is needed since each unit can arrive at its own decision. In addition, this paper proposes a curriculum multiagent deep meta-actor-critic (CMA-DMAC) algorithm, which introduces meta-reinforcement learning and curriculum learning to guide the agent training to improve the robustness and quality of the obtained SCGAC-LFC strategies. Using a simulation of the four-area LFC model for the China Southern Grid (CSG), our proposed method carries significantly lower frequency error and regulation mileage payment.

energy & fuels,construction & building technology,green & sustainable science & technology
Energy management based on multi-agent deep reinforcement learning for a multi-energy industrial park

Dafeng Zhu,Bo Yang,Yuxiang Liu,Zhaojian Wang,Kai Ma,Xinping Guan

DOI: https://doi.org/10.1016/j.apenergy.2022.118636

IF: 11.2

2022-04-01

Applied Energy

Abstract:Owing to large industrial energy consumption, industrial production has brought a huge burden to the grid in terms of renewable energy access and power supply. Due to the coupling of multiple energy sources and the uncertainty of renewable energy and demand, centralized methods require large calculation and coordination overhead. Thus, this paper proposes a multi-energy management framework achieved by decentralized execution and centralized training for an industrial park. The energy management problem is formulated as a partially-observable Markov decision process, which is intractable by dynamic programming due to the lack of the prior knowledge of the underlying stochastic process. The objective is to minimize long-term energy costs while ensuring the demand of users. To solve this issue and improve the calculation speed, a novel multi-agent deep reinforcement learning algorithm is proposed, which contains the following key points: counterfactual baseline for facilitating contributing agents to learn better policies, soft actor–critic for improving robustness and exploring optimal solutions. A novel reward is designed by Lagrange multiplier method to ensure the capacity constraints of energy storage. In addition, considering that the increase in the number of agents leads to performance degradation due to large observation spaces, an attention mechanism is introduced to enhance the stability of policy and enable agents to focus on important energy-related information, which improves the exploration efficiency of soft actor–critic. Numerical results based on actual data verify the performance of the proposed algorithm with high scalability, indicating that the industrial park can minimize energy costs under different demands.

energy & fuels,engineering, chemical
Energy Management Based on Safe Multi-Agent Reinforcement Learning for Smart Buildings in Distribution Networks

Yiyun Sun,Senlin Zhang,Meiqin Liu,Ronghao Zheng,Shanling Dong

DOI: https://doi.org/10.1016/j.enbuild.2024.114410

IF: 7.201

2024-01-01

Energy and Buildings

Abstract:The rapid urbanization and increasing use of distributed renewable energy resources have imposed a significant burden on power networks. Smart buildings equipped with artificial intelligence technology can play a pivotal role in energy management, ultimately enhancing energy efficiency and voltage quality. However, ensuring voltage stability within large-scale smart building systems presents challenges due to the coexistence of diverse energy sources and the fluctuating nature of renewable energy. This paper proposes a safe multi-energy management framework achieved by online decentralized execution and centralized training for large scale smart buildings in distribution networks. The energy management problem is formulated as a safety-augmented Markov decision process, presenting intractability for dynamic programming due to its extensive continuous state space. To solve this issue and improve the convergence speed and training process stability, a safety-augmented constrained multi-agent reinforcement learning algorithm based on reward extrapolation is proposed. In this algorithm, hazard values are introduced to enhance non-safe multi-agent reinforcement learning algorithms and meet safety constraints. A novel reward network is designed by imitating expert underlying intentions to ensure the rationality of the reward function for multi-objective tasks. Additionally, the loss function for estimating the Q-network is redesigned during training process to guarantee effective convergence. Theoretical analysis is conducted to provide the convergence guarantee. Numerical case studies based on actual data are performed to validate the effectiveness and scalability of our approach, showing that smart buildings can achieve superior energy management performance while ensuring voltage safety for distribution networks. The source code of the proposed algorithm will be available at https://github.com/SYiyun/CMARL-EX.
Large-Scale Home Energy Management Using Entropy-Based Collective Multiagent Deep Reinforcement Learning Framework

Yaodong Yang,Jianye Hao,Yan Zheng,Chao Yu

DOI: https://doi.org/10.24963/ijcai.2019/89

2019-08-01

Abstract:Smart grids are contributing to the demand-side management by integrating electronic equipment, distributed energy generation and storage and advanced meters and controllers. With the increasing adoption of electric vehicles and distributed energy generation and storage systems, residential energy management is drawing more and more attention, which is regarded as being critical to demand-supply balancing and peak load reduction. In this paper, we focus on a microgrid scenario in which modern homes interact together under a large-scale setting to better optimize their electricity cost. We first make households form a group with an economic stimulus. Then we formulate the energy expense optimization problem of the household community as a multi-agent coordination problem and present an Entropy-Based Collective Multiagent Deep Reinforcement Learning (EB-C-MADRL) framework to address it. Experiments with various real-world data demonstrate that EB-C-MADRL can reduce both the long-term group power consumption cost and daily peak demand effectively compared with existing approaches.

Federated Multi-agent Deep Reinforcement Learning for Multi-microgrid Energy Management

Multi-agent Deep Reinforcement Learning Algorithm for Distributed Economic Dispatch in Smart Grid.

Hybrid Multi-Agent Reinforcement Learning for Electric Vehicle Resilience Control Towards a Low-Carbon Transition

Multi-energy Management of Interconnected Multi-microgrid System Using Multi-agent Deep Reinforcement Learning

Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning

Multi-agent Deep Reinforcement Learning for Distributed Energy Management and Strategy Optimization of Microgrid Market

Federated Multiagent Deep Reinforcement Learning Approach via Physics-Informed Reward for Multimicrogrid Energy Management

Federated Multi-Agent Deep Reinforcement Learning Approach via Physics-Informed Reward for Multi-Microgrid Energy Management

Multi-agent deep reinforcement learning based distributed control architecture for interconnected multi-energy microgrid energy management and optimization

Multi-agent Deep Reinforcement Learning Strategy for Distributed Energy

A Multi-Agent Deep Reinforcement Learning Approach Enabled Distributed Energy Management Schedule for the Coordinate Control of Multi-Energy Hub with Gas, Electricity, and Freshwater

Distributed Energy Trading and Scheduling Among Microgrids via Multiagent Reinforcement Learning

Federated Deep Q-Network for Multiple Microgrids Energy Management

Distributed energy management of multi-area integrated energy system based on multi-agent deep reinforcement learning

Multi-agent deep reinforcement learning-based optimal energy management for grid-connected multiple energy carrier microgrids

Energy Optimization Management of Multi-microgrid Using Deep Reinforcement Learning

Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system

Multiagent deep meta reinforcement learning for sea computing-based energy management of interconnected grids considering renewable energy sources in sustainable cities

Energy management based on multi-agent deep reinforcement learning for a multi-energy industrial park

Energy Management Based on Safe Multi-Agent Reinforcement Learning for Smart Buildings in Distribution Networks

Large-Scale Home Energy Management Using Entropy-Based Collective Multiagent Deep Reinforcement Learning Framework