Abstract:The large-scale integration of distributed energy resources into the energy industry enables the fast transition to a decarbonized future but raises some potential challenges of insecure and unreliable operations. Multi-energy Microgrids (MEMGs), as localized small multi-energy systems, can effectively integrate a variety of energy components with multiple energy sectors, which have been recently recognized as a valid solution to improve the operational security and reliability. As a result, a massive amount of research has been conducted to investigate MEMG energy management problems, including both model-based optimization and model-free learning approaches. Compared to optimization approaches, reinforcement learning is being widely deployed in MEMG energy management problems owing to its ability to handle highly dynamic and stochastic processes without knowing any system knowledge. However, it is still difficult for conventional model-free reinforcement learning methods to capture the physical constraints of the MEMG model, which may therefore destroy its secure operation. To address this research challenge, this paper proposes a novel safe reinforcement learning method by learning a dynamic security assessment rule to abstract a physical-informed safety layer on top of the conventional model-free reinforcement learning energy management policy, which can respect all the physical constraints through mathematically solving an action correction formulation. In this setting, the secure energy management of the MEMG can be guaranteed for both training and test procedures. Extensive case studies based on two integrated systems (i.e., a small 6-bus power and 7-node gas network, and a large 33-bus power and 20-node gas network) are carried out to verify the superior performance of the proposed physical-informed reinforcement learning method in achieving a cost-effective MEMG energy management performance while respecting all the physical constraints, compared to conventional reinforcement learning and optimization approaches.

Physics-Shielded Multi-Agent Deep Reinforcement Learning for Safe Active Voltage Control with Photovoltaic/Battery Energy Storage Systems

Multi-Agent Reinforcement Learning with Safety Layer for Active Voltage Control

Data-Driven Multi-Agent Deep Reinforcement Learning for Distribution System Decentralized Voltage Control with High Penetration of PVs

Secure Energy Management of Multi-Energy Microgrid: A Physical-Informed Safe Reinforcement Learning Approach

Physics-Guided Multi-Agent Adversarial Reinforcement Learning for Robust Active Voltage Control With Peer-to-Peer (P2P) Energy Trading

Multi-Agent Deep Reinforcement Learning for Realistic Distribution System Voltage Control Using PV Inverters

Multi-Agent Deep Reinforcement Learning for Photovoltaics and Battery Storage Aggregators Coordinated Operation in Active Distribution Network with Incomplete Information

Decentralized Voltage Control of Large-Scale Distribution System with PVs Based on MADRL

Multi-Agent Deep Reinforcement Learning for Voltage Control with Coordinated Active and Reactive Power Optimization

Distributed Voltage Regulation of Active Distribution System Based on Enhanced Multi-agent Deep Reinforcement Learning

A Data-Driven Multi-Agent Autonomous Voltage Control Framework Using Deep Reinforcement Learning

Cooperative Multi-Agent Deep Reinforcement Learning for Adaptive Decentralized Emergency Voltage Control

Deep Reinforcement Learning Enabled Physical-Model-Free Two-Timescale Voltage Control Method for Active Distribution Systems

Two-Stage Deep Reinforcement Learning for Distribution System Voltage Regulation and Peak Demand Management

Energy Management Based on Safe Multi-Agent Reinforcement Learning for Smart Buildings in Distribution Networks

A Date-Driven Voltage Control Strategy for Distribution Network Using Deep Reinforcement Learning

A MATD3 -based Voltage Control Strategy for Distribution Networks Considering Active and Reactive Power Adjustment Costs

Three-Stage Inverter-Based Peak Shaving and Volt-VAR Control in Active Distribution Networks Using Online Safe Deep Reinforcement Learning

A Deep Reinforcement Learning Voltage Control Method for Distribution Network

Safety Constrained Multi-Agent Reinforcement Learning for Active Voltage Control

Attention Enabled Multi-agent DRL for Decentralized Volt-VAR Control of Active Distribution System Using PV Inverters and SVCs