Abstract:The integrated community energy system (ICES) has emerged as a promising solution for enhancing the efficiency of the distribution system by effectively coordinating multiple energy sources. However, the operational optimization of ICES is hindered by the physical constraints of heterogeneous networks including electricity, natural gas, and heat. These challenges are difficult to address due to the non-linearity of network constraints and the high complexity of multi-network coordination. This paper, therefore, proposes a novel Safe Reinforcement Learning (SRL) algorithm to optimize the multi-network constrained operation problem of ICES. Firstly, a comprehensive ICES model is established considering integrated demand response (IDR), multiple energy devices, and network constraints. The multi-network operational optimization problem of ICES is then presented and reformulated as a constrained Markov Decision Process (C-MDP) accounting for violating physical network constraints. The proposed novel SRL algorithm, named Primal-Dual Twin Delayed Deep Deterministic Policy Gradient (PD-TD3), solves the C-MDP by employing a Lagrangian multiplier to penalize the multi-network constraint violation, ensuring that violations are within a tolerated range and avoid over-conservative strategy with a low reward at the same time. The proposed algorithm accurately estimates the cumulative reward and cost of the training process, thus achieving a fair balance between improving profits and reducing constraint violations in a privacy-protected environment with only partial information. A case study comparing the proposed algorithm with benchmark RL algorithms demonstrates the computational performance in increasing total profits and alleviating the network constraint violations.

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are several key challenges in the techno - economic modeling and operation optimization of Integrated Community Energy Systems (ICES). Specifically: 1. **Coordination under multi - network constraints**: Most existing studies have ignored the multi - network constraints in ICES, or only considered two types of networks (such as electricity and heat), without simultaneously considering the integration of electricity, natural gas and heat networks and their constraints. Violation of these network constraints may damage the economic value of the entire community and affect the security and stability of the networks. Therefore, the paper proposes a new Multi - Network Constrained ICES (MNC - ICES) model to solve this problem. 2. **Comprehensive ICES modeling**: Existing studies often ignore some important characteristics in actual operations when modeling, such as real - world devices, the uncertainty of renewable energy, and the Integrated Demand Response (IDR) of multi - energy users. Ignoring these characteristics will lead to misleading analysis results. The paper reflects the operation logic of ICES more realistically by introducing these models. 3. **Deficiencies in optimization methods**: Current methods have deficiencies in solving non - convexity, scalability, and privacy protection. For example, mathematical programming methods require a great deal of work to achieve convexification and linearization, and require complete data information to obtain accurate solutions, which will violate privacy. Traditional reinforcement learning algorithms perform poorly when dealing with multi - network - constrained optimization problems because they lack awareness of the constraints in the optimization process. The paper proposes a state - of - the - art safe reinforcement learning algorithm (PD - TD3) based on the Lagrangian safe reinforcement learning method to solve these problems. 4. **Defects in existing safe reinforcement learning algorithms**: Existing safe reinforcement learning algorithms have multiple shortcomings, such as the difficulty in determining penalty values in the direct penalty method, the overly conservative strategy in the Lyapunov method, and the over - estimation of Q - values in the Lagrangian method. These defects lead to an unfair trade - off between rewards and costs during the training process, resulting in serious constraint violations or overly conservative strategies. The PD - TD3 algorithm proposed in the paper overcomes these defects by using a dual - network to reduce the over - estimation of action values and by delaying the update of the stable strategy and the training process of its dual variables. In summary, the main contributions of the paper are proposing a new MNC - ICES model that can consider multi - network constraints, real - world devices, the uncertainty of renewable energy, and the demand response of multi - energy users, and developing an advanced safe reinforcement learning algorithm (PD - TD3) to solve the operation optimization problem under multi - network constraints, thereby improving the operation efficiency and security of the Integrated Community Energy Systems.

Techno-Economic Modeling and Safe Operational Optimization of Multi-Network Constrained Integrated Community Energy Systems

Multi-Network Constrained Operational Optimization in Community Integrated Energy Systems: A Safe Reinforcement Learning Approach

Secure Energy Management of Multi-Energy Microgrid: A Physical-Informed Safe Reinforcement Learning Approach

Online Operational Decision-making for Integrated Electric-Gas Systems with Safe Reinforcement Learning

An Integrated Demand Response-Based Energy Management Strategy for Integrated Energy System Based on Deep Reinforcement Learning

Rethinking Safe Policy Learning for Complex Constraints Satisfaction: A Glimpse in Real-Time Security Constrained Economic Dispatch Integrating Energy Storage Units

Dynamic optimization of an integrated energy system with carbon capture and power-to-gas interconnection: A deep reinforcement learning-based scheduling strategy

Coordinated energy management for integrated energy system incorporating multiple flexibility measures of supply and demand sides: A deep reinforcement learning approach

Towards Pareto-optimal energy management in integrated energy systems: A multi-agent and multi-objective deep reinforcement learning approach

Optimal Operation of Integrated Energy System Based on Deep Reinforcement Learning

Local Integrated Energy System Operational Optimization Considering Multi-Type Uncertainties: A Reinforcement Learning Approach Based on Improved TD3 Algorithm

Optimal Scheduling of Integrated Demand Response-Enabled Community Integrated Energy Systems in Uncertain Environments

Safe Imitation Learning-based Optimal Energy Storage Systems Dispatch in Distribution Networks

Deep Reinforcement Learning-driven Cross-Community Energy Interaction Optimal Scheduling

A Hybrid Data-Driven Method for Low-Carbon Economic Energy Management Strategy in Electricity-Gas Coupled Energy Systems Based on Transformer Network and Deep Reinforcement Learning

Distributed optimization of electricity-Gas-Heat integrated energy system with multi-agent deep reinforcement learning

Low-carbon Economic Dispatch of Electricity-Heat-Gas Integrated Energy Systems Based on Deep Reinforcement Learning

Integrated Energy Hub Dispatch with a Multi-Mode CAES–BESS Hybrid System: an Option-Based Hierarchical Reinforcement Learning Approach

A Multi-Agent Deep Constrained Q-Learning Method for Smart Building Energy Management Under Uncertainties

Integrated Energy Cluster Hierarchical Regulation Technology Considering Demand Response

Network Resource Allocation Algorithm Using Reinforcement Learning Policy-Based Network in a Smart Grid Scenario