Abstract:Automatic peer-to-peer energy trading can be defined as a Markov decision process and designed using deep reinforcement learning. We consider prosumer as an entity that consumes and produces electric energy with an energy storage system, and define the prosumer’s objective as maximizing the profit through participation in peer-to-peer energy trading, similar to that of the agents in stock trading. In this paper, we propose an automatic peer-to-peer energy trading model by adopting a deep Q-network-based automatic trading algorithm originally designed for stock trading. Unlike in stock trading, the assets held by a prosumer may change owing to factors such as the consumption and generation of energy by the prosumer in addition to the changes from trading activities. Therefore, we propose a new trading evaluation criterion that considers these factors by defining profit as the sum of the gains from four components: electricity bill, trading, electric energy stored in the energy storage system, and virtual loss. For the proposed automatic peer-to-peer energy trading algorithm, we adopt a long-term delayed reward method that evaluates the delayed reward that occurs once per month by generating the termination point of an episode at each month and propose a long short-term delayed reward method that compensates for the issue with the long-term delayed reward method having only a single evaluation per month. This long short-term delayed reward method enables effective learning of the monthly long-term trading patterns and the short-term trading patterns at the same time, leading to a better trading strategy. The experimental results showed that the long short-term delayed reward method-based energy trading model achieves higher profits every month both in the progressive and fixed rate systems throughout the year and that prosumer participating in the trading not only earns profits every month but also reduces loss from over-generation of electric energy in the case of South Korea. Further experiments with various progressive rate systems of Japan, Taiwan, and the United States as well as in different prosumer environments indicate the general applicability of the proposed method.

A Deep Reinforcement Learning-Based Bidding Strategy for Participants in a Peer-to-peer Energy Trading Scenario

Deep Reinforcement Learning for Strategic Bidding in Electricity Markets

Deep reinforcement learning-based optimal bidding strategy for real-time multi-participant electricity market with short-term load

Strategic Peer-to-peer Energy Trading Framework Considering Distribution Network Constraints

Peer-to-Peer Energy Transactions for Prosumers Based on Improved Deep Deterministic Policy Gradient Algorithm

P2P Energy Trading through Prospect Theory, Differential Evolution, and Reinforcement Learning

Peer-to-Peer Trading for Energy-Saving Based on Reinforcement Learning

Surrogate model enabled deep reinforcement learning for hybrid energy community operation

Automatic P2P Energy Trading Model Based on Reinforcement Learning Using Long Short-Term Delayed Reward

An Integrated Demand Response-Based Energy Management Strategy for Integrated Energy System Based on Deep Reinforcement Learning

Prospect Theory-inspired Automated P2P Energy Trading with Q-learning-based Dynamic Pricing

A decentralized peer-to-peer energy trading strategy considering flexible resource involvement and renewable energy uncertainty

A Scalable Privacy-Preserving Multi-Agent Deep Reinforcement Learning Approach for Large-Scale Peer-to-Peer Transactive Energy Trading

Multi-Agent Reinforcement Learning for Automated Peer-to-Peer Energy Trading in Double-Side Auction Market

Dynamic Pricing Strategy in Electricity Trading Market Based on Reinforcement Learning

A Hierarchical Deep Reinforcement Learning-Based Community Energy Trading Scheme for a Neighborhood of Smart Households

Deep Reinforcement Learning-Based Trading Strategy for Load Aggregators on Price-Responsive Demand

Peer-to-peer energy trading optimization in energy communities using multi-agent deep reinforcement learning

Multi-agent deep deterministic policy gradient algorithm for peer-to-peer energy trading considering distribution network constraints

Energy Trading in Smart Grid: A Deep Reinforcement Learning-based Approach

Peer-to-peer energy trading with energy trading consistency in interconnected multi-energy microgrids: A multi-agent deep reinforcement learning approach