Abstract:The large-scale integration of distributed energy resources into the energy industry enables the fast transition to a decarbonized future but raises some potential challenges of insecure and unreliable operations. Multi-energy Microgrids (MEMGs), as localized small multi-energy systems, can effectively integrate a variety of energy components with multiple energy sectors, which have been recently recognized as a valid solution to improve the operational security and reliability. As a result, a massive amount of research has been conducted to investigate MEMG energy management problems, including both model-based optimization and model-free learning approaches. Compared to optimization approaches, reinforcement learning is being widely deployed in MEMG energy management problems owing to its ability to handle highly dynamic and stochastic processes without knowing any system knowledge. However, it is still difficult for conventional model-free reinforcement learning methods to capture the physical constraints of the MEMG model, which may therefore destroy its secure operation. To address this research challenge, this paper proposes a novel safe reinforcement learning method by learning a dynamic security assessment rule to abstract a physical-informed safety layer on top of the conventional model-free reinforcement learning energy management policy, which can respect all the physical constraints through mathematically solving an action correction formulation. In this setting, the secure energy management of the MEMG can be guaranteed for both training and test procedures. Extensive case studies based on two integrated systems (i.e., a small 6-bus power and 7-node gas network, and a large 33-bus power and 20-node gas network) are carried out to verify the superior performance of the proposed physical-informed reinforcement learning method in achieving a cost-effective MEMG energy management performance while respecting all the physical constraints, compared to conventional reinforcement learning and optimization approaches.

Online Operational Decision-making for Integrated Electric-Gas Systems with Safe Reinforcement Learning

Secure Energy Management of Multi-Energy Microgrid: A Physical-Informed Safe Reinforcement Learning Approach

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

A Safe DRL Method for Fast Solution of Real-Time Optimal Power Flow

Safe reinforcement learning method integrating process knowledge for real-time scheduling of gas supply network

Real-Time Optimal Power Flow Method Via Safe Deep Reinforcement Learning Based on Primal-Dual and Prior Knowledge Guidance

A Deep Reinforcement Learning Based Approach for Optimal Active Power Dispatch

Multi-Network Constrained Operational Optimization in Community Integrated Energy Systems: A Safe Reinforcement Learning Approach

Data-Driven Online Energy Scheduling of a Microgrid Based on Deep Reinforcement Learning

Safe deep reinforcement learning for real-time AC optimal power flow: A near-optimal solution

Soft Actor-Critic DRL Algorithm for Interval Optimal Dispatch of Integrated Energy Systems with Uncertainty in Demand Response and Renewable Energy

Techno-Economic Modeling and Safe Operational Optimization of Multi-Network Constrained Integrated Community Energy Systems

On Training Effective Reinforcement Learning Agents for Real-time Power Grid Operation and Control

A Human-Machine Reinforcement Learning Method for Cooperative Energy Management.

Towards Pareto-optimal energy management in integrated energy systems: A multi-agent and multi-objective deep reinforcement learning approach

Safe Deep Reinforcement Learning for Microgrid Energy Management in Distribution Networks with Leveraged Spatial–Temporal Perception

Deep Reinforcement Learning–based Approach for Optimizing Energy Conversion in Integrated Electrical and Heating System with Renewable Energy

Energy management strategy via maximum entropy reinforcement learning for an extended range logistics vehicle

A Low-Carbon Economic Dispatch Method for Power Systems with Carbon Capture Plants Based on Safe Reinforcement Learning

Rethinking Safe Policy Learning for Complex Constraints Satisfaction: A Glimpse in Real-Time Security Constrained Economic Dispatch Integrating Energy Storage Units

Real-time Energy Management of a Low-Carbon Micro-Grid Based on Deep Reinforcement Learning