Abstract:Energy management systems (EMS) are becoming increasingly important in order to utilize the continuously growing curtailed renewable energy. Promising energy storage systems (ESS), such as batteries and green hydrogen should be employed to maximize the efficiency of energy stakeholders. However, optimal decision-making, i.e., planning the leveraging between different strategies, is confronted with the complexity and uncertainties of large-scale problems. Here, we propose a sophisticated deep reinforcement learning (DRL) methodology with a policy-based algorithm to realize the real-time optimal ESS planning under the curtailed renewable energy uncertainty. A quantitative performance comparison proved that the DRL agent outperforms the scenario-based stochastic optimization (SO) algorithm, even with a wide action and observation space. Owing to the uncertainty rejection capability of the DRL, we could confirm a robust performance, under a large uncertainty of the curtailed renewable energy, with a maximizing net profit and stable system. Action-mapping was performed for visually assessing the action taken by the DRL agent according to the state. The corresponding results confirmed that the DRL agent learns the way like what a human expert would do, suggesting reliable application of the proposed methodology.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use deep reinforcement learning (DRL) to achieve optimal real - time planning for hybrid energy storage systems (such as battery energy storage systems BESS and green hydrogen production systems AWE) for curtailed renewable energy under conditions with a large amount of uncertainty. Specifically, the research aims to overcome the limitations of traditional mathematical programming methods (such as stochastic optimization SO of scenario analysis) such as high computational complexity and difficulty in scaling when dealing with large - scale problems through deep reinforcement learning methods, thereby improving the efficiency and economy of energy management systems. ### Background and Objectives of the Paper With the rapid increase in the installed capacity of renewable energy (such as wind and solar energy), the problem of power surplus due to supply - demand imbalance is becoming increasingly serious. In order to meet this challenge, curtailment (i.e., reducing the power generation to below the original design capacity) measures are usually taken to control the surplus power. However, this approach will still lead to energy shortages during peak hours. Therefore, integrating the curtailed renewable energy into the energy storage system, rather than adding more power generation equipment, is beneficial from both economic and environmental perspectives. Traditional energy management planning problems are usually solved using mathematical programming (MP) methods, but this method has limitations when dealing with renewable energy with high uncertainty. The paper proposes a method based on deep reinforcement learning (DRL) for real - time decision - making on how to store the curtailed renewable energy in BESS or convert it into hydrogen through AWE to maximize the profit within the entire operation cycle (such as one year) and ensure the stable operation of the system. ### Main Contributions - **Proposed a DRL - based real - time planning AI**: This model can make optimal decisions according to the current state (such as electricity price, hydrogen price, state of charge of the energy storage system, etc.), and decide how much energy to store in BESS and how much energy to convert into hydrogen through AWE. - **Performance verification**: By comparing with the stochastic optimization (SO) algorithm considering uncertainty, the superior performance of the DRL model in dealing with large - scale uncertainty is proved. - **Enhanced interpretability**: By using the action mapping diagram to intuitively show the actions taken by the DRL agent in different states, human experts can understand and accept the decision - making logic of the DRL model. ### Conclusions The paper verifies the effectiveness and robustness of the proposed DRL model in dealing with renewable energy storage planning problems through experiments, especially its performance is better than traditional methods when facing high uncertainty. In addition, this research also emphasizes the interpretability and reliability of the DRL model in practical applications, providing new ideas for the design of future energy management systems.

Optimal Planning of Hybrid Energy Storage Systems using Curtailed Renewable Energy through Deep Reinforcement Learning

A Constraint Enforcement Deep Reinforcement Learning Framework for Optimal Energy Storage Systems Dispatch

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Optimal Energy System Scheduling Using A Constraint-Aware Reinforcement Learning Algorithm

Optimal Coordination of Distributed Energy Resources Using Deep Deterministic Policy Gradient

Deep Reinforcement Learning Based Optimal Energy Management of Multi-energy Microgrids with Uncertainties

An Intelligent Energy Management Strategy for Hybrid Vehicle with Irrational Actions Using Twin Delayed Deep Deterministic Policy Gradient

Physical model-assisted deep reinforcement learning for energy management optimization of industrial electric-hydrogen coupling system with hybrid energy storage

A Holistic Power Optimization Approach for Microgrid Control Based on Deep Reinforcement Learning

Interior-point policy optimization based multi-agent deep reinforcement learning method for secure home energy management under various uncertainties

A Control Strategy Based on Deep Reinforcement Learning Under the Combined Wind-Solar Storage System

Optimal dispatch of an energy hub with compressed air energy storage: A safe reinforcement learning approach

Data-Driven Online Energy Scheduling of a Microgrid Based on Deep Reinforcement Learning

Energy Management of Smart Home with Home Appliances, Energy Storage System and Electric Vehicle: A Hierarchical Deep Reinforcement Learning Approach

A Deep Reinforcement Learning Based Approach for Optimal Active Power Dispatch

Attention and masking embedded ensemble reinforcement learning for smart energy optimization and risk evaluation under uncertainties

Deep Reinforcement Learning Based Energy Management of Hybrid Electric Vehicles with Expert Knowledge

Deep Reinforcement Learning Based Energy Scheduling of a Hybrid Electricity/Heat/Hydrogen Energy System

Deep Reinforcement Learning for Stochastic Dynamic Microgrid Energy Management

Energy management strategy via maximum entropy reinforcement learning for an extended range logistics vehicle

Optimal Scheduling in IoT-Driven Smart Isolated Microgrids Based on Deep Reinforcement Learning