Abstract:As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.

Deep Reinforcement Learning Based Pricing Strategy of Aggregators Considering Renewable Energy

Deep Reinforcement Learning for Strategic Bidding in Electricity Markets

Deep Reinforcement Learning-Based Trading Strategy for Load Aggregators on Price-Responsive Demand

ReLeDP: Reinforcement-Learning-Assisted Dynamic Pricing for Wireless Smart Grid

Dynamic Demand-Aware Power Grid Intelligent Pricing Algorithm Based on Deep Reinforcement Learning

Duck Curve Aware Dynamic Pricing and Battery Scheduling Strategy Using Reinforcement Learning

Bi-level stochastic real-time pricing model in multi-energy generation system: A reinforcement learning approach

A distributed real-time pricing strategy based on reinforcement learning approach for smart grid

Finding individual strategies for storage units in electricity market models using deep reinforcement learning

Energy Management of Networked Microgrids with Real-Time Pricing by Reinforcement Learning

A Control Strategy Based on Deep Reinforcement Learning Under the Combined Wind-Solar Storage System

Dynamic Incentive Pricing on Charging Stations for Real-Time Congestion Management in Distribution Network: An Adaptive Model-Based Safe Deep Reinforcement Learning Method

Price-Matching-Based Regional Energy Market with Hierarchical Reinforcement Learning Algorithm

An application of reinforcement learning to residential energy storage under real-time pricing

Demand management and pricing in renewable integrated plug-in electric vehicle charging station using reinforcement learning

Arbitrage of Energy Storage in Electricity Markets with Deep Reinforcement Learning

Blockchain-Assisted Secure Energy Trading in Electricity Markets: A Tiny Deep Reinforcement Learning-Based Stackelberg Game Approach

A Reinforcement Learning-Based Decision System for Electricity Pricing Plan Selection by Smart Grid End Users.

Optimal demand response based dynamic pricing strategy via Multi-Agent Federated Twin Delayed Deep Deterministic policy gradient algorithm

Influence of Discrete and Continuous Action Spaces on Deep Reinforcement Learning-Based Pricing Strategy Optimization for Electricity Retailers

An optimal solutions-guided deep reinforcement learning approach for online energy storage control