Abstract:Automatic peer-to-peer energy trading can be defined as a Markov decision process and designed using deep reinforcement learning. We consider prosumer as an entity that consumes and produces electric energy with an energy storage system, and define the prosumer’s objective as maximizing the profit through participation in peer-to-peer energy trading, similar to that of the agents in stock trading. In this paper, we propose an automatic peer-to-peer energy trading model by adopting a deep Q-network-based automatic trading algorithm originally designed for stock trading. Unlike in stock trading, the assets held by a prosumer may change owing to factors such as the consumption and generation of energy by the prosumer in addition to the changes from trading activities. Therefore, we propose a new trading evaluation criterion that considers these factors by defining profit as the sum of the gains from four components: electricity bill, trading, electric energy stored in the energy storage system, and virtual loss. For the proposed automatic peer-to-peer energy trading algorithm, we adopt a long-term delayed reward method that evaluates the delayed reward that occurs once per month by generating the termination point of an episode at each month and propose a long short-term delayed reward method that compensates for the issue with the long-term delayed reward method having only a single evaluation per month. This long short-term delayed reward method enables effective learning of the monthly long-term trading patterns and the short-term trading patterns at the same time, leading to a better trading strategy. The experimental results showed that the long short-term delayed reward method-based energy trading model achieves higher profits every month both in the progressive and fixed rate systems throughout the year and that prosumer participating in the trading not only earns profits every month but also reduces loss from over-generation of electric energy in the case of South Korea. Further experiments with various progressive rate systems of Japan, Taiwan, and the United States as well as in different prosumer environments indicate the general applicability of the proposed method.

Peer-to-Peer Energy Transactions for Prosumers Based on Improved Deep Deterministic Policy Gradient Algorithm

Multi-agent deep deterministic policy gradient algorithm for peer-to-peer energy trading considering distribution network constraints

Agent-Based Energy Sharing Mechanism Using Deep Deterministic Policy Gradient Algorithm

A peer‐to‐peer joint energy and reserve market considering renewable generation uncertainty: A generalized Nash equilibrium approach

Strategic Peer-to-peer Energy Trading Framework Considering Distribution Network Constraints

P2P Energy Trading through Prospect Theory, Differential Evolution, and Reinforcement Learning

Mean-Field Multi-Agent Reinforcement Learning for Peer-to-Peer Multi-Energy Trading

Peer-to-Peer Energy Trading and Energy Conversion in Interconnected Multi-Energy Microgrids Using Multi-Agent Deep Reinforcement Learning

A Scalable Privacy-Preserving Multi-Agent Deep Reinforcement Learning Approach for Large-Scale Peer-to-Peer Transactive Energy Trading

Peer-to-peer energy trading with energy trading consistency in interconnected multi-energy microgrids: A multi-agent deep reinforcement learning approach

Optimal Prosumer Operation with Consideration for Bounded Rationality in Peer-to-Peer Energy Trading Systems

Online Optimization for Real-Time Peer-to-Peer Electricity Market Mechanisms

Prospect Theory-inspired Automated P2P Energy Trading with Q-learning-based Dynamic Pricing

Multi-Agent Reinforcement Learning for Automated Peer-to-Peer Energy Trading in Double-Side Auction Market

Peer-to-Peer Energy Trading Using Prediction Intervals of Renewable Energy Generation.

Peer-to-peer energy trading optimization in energy communities using multi-agent deep reinforcement learning

A decentralized peer-to-peer energy trading strategy considering flexible resource involvement and renewable energy uncertainty

Peer-to-Peer Trading for Energy-Saving Based on Reinforcement Learning

Optimal Peer-to-Peer Energy Trading Under Load Uncertainty Incorporating Carbon Emission and Transaction Cost for Grid-Connected Prosumers

Automatic P2P Energy Trading Model Based on Reinforcement Learning Using Long Short-Term Delayed Reward

Peer-to-peer energy trading optimization for community prosumers considering carbon cap-and-trade