Networked Multiagent Safe Reinforcement Learning for Low-carbon Demand Management in Distribution Network

Jichen Zhang,Linwei Sang,Yinliang Xu,Hongbin Sun

DOI: https://doi.org/10.1109/TSTE.2024.3355123

2023-11-27

Abstract:This paper proposes a multiagent based bi-level operation framework for the low-carbon demand management in distribution networks considering the carbon emission allowance on the demand side. In the upper level, the aggregate load agents optimize the control signals for various types of loads to maximize the profits; in the lower level, the distribution network operator makes optimal dispatching decisions to minimize the operational costs and calculates the distribution locational marginal price and carbon intensity. The distributed flexible load agent has only incomplete information of the distribution network and cooperates with other agents using networked communication. Finally, the problem is formulated into a networked multi-agent constrained Markov decision process, which is solved using a safe reinforcement learning algorithm called consensus multi-agent constrained policy optimization considering the carbon emission allowance for each agent. Case studies with the IEEE 33-bus and 123-bus distribution network systems demonstrate the effectiveness of the proposed approach, in terms of satisfying the carbon emission constraint on demand side, ensuring the safe operation of the distribution network and preserving privacy of both sides.

Systems and Control,Artificial Intelligence

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is: in the distribution network, how to achieve low - carbon demand management through multi - agent systems while ensuring the safe operation of the system and protecting user privacy. Specifically, the paper proposes a two - layer operation framework based on multi - agent systems, aiming at: 1. **Maximizing the profit of the aggregated load**: At the upper layer, the aggregated load agent optimizes the control signals under the limitation of carbon emission quotas to maximize the profit of various types of loads. 2. **Minimizing the operating cost of the distribution network**: At the lower layer, the distribution network operator minimizes the operating cost through optimal scheduling decisions and calculates the distribution location marginal price and carbon intensity. 3. **Protecting privacy**: In the multi - agent system, each agent has only partial information and cooperates with other agents through network communication to ensure the protection of privacy between users and system operators during the optimization process. 4. **Adapting to uncertain renewable energy resources**: The proposed framework can adapt to uncertain renewable energy resources and ensure the flexibility and robustness of the system. To achieve the above goals, the paper models the problem as a networked multi - agent constrained Markov decision process (NMACMDP) and uses a safe reinforcement learning algorithm called Consensus Multi - Agent Constrained Policy Optimization (CMACPO) to solve the problem. This method can optimize load management and the operation of the distribution network while meeting carbon emission limitations.

Networked Multiagent Safe Reinforcement Learning for Low-carbon Demand Management in Distribution Network

Networked Multiagent-Based Safe Reinforcement Learning for Low-Carbon Demand Management in Distribution Networks

Multi-Agent Reinforcement Learning-based Distributed Economic Dispatch Considering Network attacks and Uncertain Costs

Multi-agent Deep Reinforcement Learning Algorithm for Distributed Economic Dispatch in Smart Grid.

Multi-service Provision for Electric Vehicles in Power-Transportation Networks Towards a Low-Carbon Transition: A Hierarchical and Hybrid Multi-Agent Reinforcement Learning Approach

Target-Value-Competition-Based Multi-Agent Deep Reinforcement Learning Algorithm for Distributed Nonconvex Economic Dispatch

Hybrid Multi-Agent Reinforcement Learning for Electric Vehicle Resilience Control Towards a Low-Carbon Transition

Energy Management Based on Safe Multi-Agent Reinforcement Learning for Smart Buildings in Distribution Networks

Security Constrained Dispatch for Renewable Proliferated Distribution Network Based on Safe Reinforcement Learning

A Cloud-Edge Collaboration Solution for Distribution Network Reconfiguration Using Multi-Agent Deep Reinforcement Learning

Multiagent-Based Reinforcement Learning for Optimal Reactive Power Dispatch.

Electric Vehicles Management in Distribution Network: A Data-Efficient Bi-level Safe Deep Reinforcement Learning Method

Deep reinforcement learning based multi-level dynamic reconfiguration for urban distribution network: A cloud-edge collaboration architecture

An Augmented Lagrangian-based Safe Reinforcement Learning Algorithm for Carbon-Oriented Optimal Scheduling of EV Aggregators

Collaborative operation optimization of distribution system and virtual power plants using multi‐agent deep reinforcement learning with parameter‐sharing mechanism

Optimal Bi-Level Bidding and Dispatching Strategy between Active Distribution Network and Virtual Alliances Using Distributed Robust Multi-Agent Deep Reinforcement Learning

Distributed Multiagent Reinforcement Learning With Action Networks for Dynamic Economic Dispatch

Multi-Network Constrained Operational Optimization in Community Integrated Energy Systems: A Safe Reinforcement Learning Approach

Real-time Operation Optimization in Active Distribution Networks Based on Multi-Agent Deep Reinforcement Learning

AC/DC hybrid distribution network reconfiguration with microgrid formation using multi-agent soft actor-critic

A Multi-Agent Reinforcement Learning Method for Distribution System Restoration Considering Dynamic Network Reconfiguration