A Safe Energy Policy Optimization Method for Multienergy Microgrid Control

Yigong Zhang,Qiushi Cui,Lixian Shi,Yang Weng,Jian Li

DOI: https://doi.org/10.1109/tpwrs.2024.3421272

IF: 7.326

2024-01-01

IEEE Transactions on Power Systems

Abstract:Multi-energy microgrid (MEMG) control plays a crucial role in meeting diverse energy demands on the user side. Traditional methods face challenges in handling renewable energy sources (RES) uncertainty, as it is difficult to accurately model uncertainty in the numerical solution process. Recent researches address uncertainty by employing Markov decision process theory to develop deep reinforcement learning-based MEMG control methods, aiming to derive policies as opposed to numerical solutions. However, the complex MEMG environment and system constraints limit their actual deployment. Therefore, we reformulate MEMG control as a constrained Markov decision process and solve it by a proposed safe energy policy optimization (SEPO). Specifically, facing the challenge of a high variance MEMG environment caused by load fluctuation and RES uncertainty, a RoSi activation function is designed to ensure robust learning. Then, to satisfy MEMG system constraints, a Lagrangian multiplier is designed to decouple learning of constraints from the pursuit of the reward function of economic objective. Lastly, the difficult-to-satisfy equation-type constraint of heat power in MEMG is converted into the easy-to-satisfy threshold-type constraint through a pre-compensation reward function, which extends the feasible action domain from hyperplane to hyperspace. Case studies validate the superior performance of SEPO in reducing MEMG economic costs, ensuring learning robustness, and guaranteeing control security.