Abstract:Energy hub (EH) is an independent entity that benefits to the efficiency, flexibility, and reliability of integrated energy systems (IESs). On the other hand, the rapid emerging of electric vehicles (EVs) drives the EV aggregator (EVAGG) as another independent entity to facilitate the electricity exchange with the grid. However, due to privacy consideration for different owners, it is challenging to investigate the optimal coordinated strategies for such interconnected entities only by exchanging the information of electrical energy. Besides, the existence of parameter uncertainties (load demands, EVs' charging behaviors, wind power and photovoltaic generation), continuous decision space, dynamic energy flows, and non-convex multi-objective function is difficult to solve. To this end, this paper proposes a novel model-free multi-agent deep reinforcement learning (MADRL)-based decentralized coordination model to minimize the energy costs of EH entities and maximize profits of EVAGGs. First, a long short-term memory (LSTM) module is used to capture the future trend of uncertainties. Then, the coordination problem is formulated as Markov games and solved by the attention enabled MADRL algorithm, where the EH or EVAGG entity is modeled as an adaptive agent. An attention mechanism makes each agent only focus on state information related to the reward. The proposed MADRL adopts the forms of offline centralized training to learn the optimal coordinated control strategy, and decentralized execution to enable agents' online decisions to only require local measurements. A safety network is employed to cope with equality constraints (demand-supply balance). Simulation results illustrate that the proposed method achieves similar results compared to the traditional model-based method with perfect knowledge of system models, and the computation performance is at least two orders of magnitudes shorter than the traditional method. The testing results of the proposed method are better than those of the Concurrent and other MADRL method, with 10.79%/3.06% lower energy cost and 17.11%/6.82% higher profits of aggregator. Besides, the electric equality constraint of the proposed method is only 0.25 MW averaged per day, which is a small and acceptable violation.

Multi-Agent Deep Reinforcement Learning for Coordinated Energy Trading and Flexibility Services Provision in Local Electricity Markets

Deep Reinforcement Learning for Strategic Bidding in Electricity Markets

LEM for DERs and Flexible Loads

Establishing a hierarchical local market structure using multi-cut Benders decomposition

Decentralized Coordination of Distributed Energy Resources through Local Energy Markets and Deep Reinforcement Learning

Sequential Clearing of Network-aware Local Energy and Flexibility Markets in Community-based Grids

A Peer-To-Peer Market Mechanism for Distributed Energy Resourses

Mean-Field Multi-Agent Reinforcement Learning for Peer-to-Peer Multi-Energy Trading

A Multi-Agent Deep Reinforcement Learning Approach for a Distributed Energy Marketplace in Smart Grids

Indirect Multi-energy Transactions of Energy Internet with Deep Reinforcement Learning Approach

Two-timescale autonomous energy management strategy based on multi-agent deep reinforcement learning approach for residential multicarrier energy system

Price-Matching-Based Regional Energy Market with Hierarchical Reinforcement Learning Algorithm

Federated reinforcement learning for smart building joint peer-to-peer energy and carbon allowance trading

Peer-to-Peer Energy Trading of Solar and Energy Storage: A Networked Multiagent Reinforcement Learning Approach

Peer-to-peer energy trading with energy trading consistency in interconnected multi-energy microgrids: A multi-agent deep reinforcement learning approach

Multi-agent energy management optimization for integrated energy systems under the energy and carbon co-trading market

A Competitive Framework for the Participation of Multi-Microgrids in the Community Energy Trading Market: A Case Study

Novel Data-Driven Decentralized Coordination Model for Electric Vehicle Aggregator and Energy Hub Entities in Multi-Energy System Using an Improved Multi-Agent DRL Approach

Deep Multi-Task Multi-Agent Reinforcement Learning Based Joint Bidding and Pricing Strategy of Price-Maker Load Serving Entity

A Hierarchical Deep Reinforcement Learning-Based Community Energy Trading Scheme for a Neighborhood of Smart Households

A Local Energy Market for Electricity and Hydrogen