Abstract:The global energy landscape is undergoing a transformation towards decarbonization, sustainability, and cost-efficiency. In this transition, microgrid systems integrated with renewable energy sources (RES) and energy storage systems (ESS) have emerged as a crucial component. However, optimizing the operational control of such an integrated energy system lacks a holistic view of multiple environmental, infrastructural and economic considerations, not to mention the need to factor in the uncertainties from both the supply and demand. This paper presents a holistic datadriven power optimization approach based on deep reinforcement learning (DRL) for microgrid control considering the multiple needs of decarbonization, sustainability and cost-efficiency. First, two data-driven control schemes, namely the prediction-based (PB) and prediction-free (PF) schemes, are devised to formulate the control problem within a Markov decision process (MDP). Second, a multivariate objective (reward) function is designed to account for the market profits, carbon emissions, peak load, and battery degradation of the microgrid system. Third, we develop a Double Dueling Deep Q Network (D3QN) architecture to optimize the power flows for real-time energy management and determine charging/discharging strategies of ESS. Finally, extensive simulations are conducted to demonstrate the effectiveness and superiority of the proposed approach through a comparative analysis. The results and analysis also suggest the respective circumstances for using the two control schemes in practical implementations with uncertainties.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to optimize the operation control of micro - grids integrating renewable energy sources (RES) and energy storage systems (ESS) in micro - grid control while considering environmental, infrastructure and economic goals, especially in the face of supply - demand uncertainties. Specifically, the paper aims to propose a comprehensive data - driven power optimization method through deep reinforcement learning (DRL) to achieve a balance between decarbonization, sustainability and cost - effectiveness of the micro - grid system. ### Background of the Paper and Problem Definition With the continuous growth of energy demand and the intensification of energy shortages, micro - grids are moving towards a cleaner and more sustainable direction. Currently, micro - grids integrating renewable energy sources (such as photovoltaic and wind power generation) and energy storage systems play a key role in the energy system transformation because they provide higher security, reliability and cleanliness. However, the optimization of the operation control of such integrated energy systems lacks comprehensive consideration of environmental, infrastructure and economic goals, especially in the context where supply - demand uncertainties need to be considered. ### Research Objectives To meet the above challenges, the paper proposes a comprehensive power optimization method for micro - grid control based on deep reinforcement learning (DRL). The main contributions include: 1. **Unified DRL Framework**: A unified DRL framework for micro - grid control is designed, in which a new objective (reward) function is introduced to balance the decarbonization, sustainability and cost - effectiveness performance of the micro - grid system. 2. **Data - Driven D3QN Architecture**: A data - driven double - dueling deep Q - network (D3QN) architecture is developed to optimize the power management of micro - grid systems integrating RES and ESS. 3. **Two Control Schemes**: Two control schemes (i.e., prediction - based scheme and non - prediction scheme) are designed within the unified DRL framework to utilize the learning ability of D3QN to operate energy storage devices in different situations. 4. **Extensive Simulation**: Extensive simulations are carried out to demonstrate the effectiveness of the proposed method and to suggest different situations of using the two control schemes when uncertainties occur in practical applications. ### Key Technologies and Methods - **Micro - grid Modeling**: The power balance problem of the micro - grid and its constraints are defined, including the power imported from the national grid, battery charge - discharge power, renewable energy generation and total consumer demand. - **Deep Reinforcement Learning**: Classic DRL algorithms, especially DQN, DDQN and Dueling DQN, are introduced, and how these algorithms are applied to micro - grid control is explained. - **D3QN Architecture**: Combining the advantages of DDQN and Dueling DQN, a D3QN architecture is proposed to optimize the real - time energy management of micro - grids. - **Reward Function**: A comprehensive reward function is designed, taking into account factors such as operating costs, carbon emissions, peak load and battery degradation. ### Conclusion Through the above methods, the paper shows how to achieve multi - objective optimization through deep reinforcement learning techniques in a complex micro - grid environment, thereby improving the operation efficiency and sustainability of micro - grids.

A Holistic Power Optimization Approach for Microgrid Control Based on Deep Reinforcement Learning

A Bayesian Deep Reinforcement Learning-Based Resilient Control for Multi-Energy Micro-Gird

Double Deep Q-learning Based Real-Time Optimization Strategy for Microgrids

Data-Driven Online Energy Scheduling of a Microgrid Based on Deep Reinforcement Learning

Deep Reinforcement Learning Based Double-layer Optimization Method for Energy Management of Microgrid

Deep Reinforcement Learning Microgrid Optimization Strategy Considering Priority Flexible Demand Side

Deep Reinforcement Learning Based Optimal Energy Management of Multi-energy Microgrids with Uncertainties

Real-Time Energy Management of a Microgrid Using Deep Reinforcement Learning

Deep Reinforcement Learning-Based Method for Joint Optimization of Mobile Energy Storage Systems and Power Grid with High Renewable Energy Sources

Optimizing Load Scheduling in Power Grids Using Reinforcement Learning and Markov Decision Processes

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

A Control Strategy Based on Deep Reinforcement Learning Under the Combined Wind-Solar Storage System

Energy Scheduling and Decision Learning of Combined Cooling, Heating and Power Microgrid Based on Deep Deterministic Policy Gradient

Data-Efficient Deep Reinforcement Learning-Based Optimal Generation Control in DC Microgrids

Deep Reinforcement Learning–based Approach for Optimizing Energy Conversion in Integrated Electrical and Heating System with Renewable Energy

Deep Reinforcement Learning for Microgrid Cost Optimization Considering Load Flexibility

Deep Reinforcement Learning Based Bi-layer Optimal Scheduling for Microgrids Considering Flexible Load Control

Power balance control of RES integrated power system by deep reinforcement learning with optimized utilization rate of renewable energy

Parametric Dueling DQN- and DDPG-Based Approach for Optimal Operation of Microgrids

Parallel Hybrid Deep Reinforcement Learning for Real-Time Energy Management of Microgrid

Deep Reinforcement Learning for Stochastic Dynamic Microgrid Energy Management