Abstract:As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.

Developing Optimal Energy Arbitrage Strategy for Energy Storage System Using Reinforcement Learning

Arbitrage of Energy Storage in Electricity Markets with Deep Reinforcement Learning

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Energy Storage Arbitrage in Real-Time Markets via Reinforcement Learning

Reinforcement Learning-Enhanced Adaptive Scheduling of Battery Energy Storage Systems in Energy Markets

Interpretable Deep Reinforcement Learning for Optimizing Heterogeneous Energy Storage Systems

Routing and Scheduling of Mobile Energy Storage System for Electricity Arbitrage Based on Two-Layer Deep Reinforcement Learning

Optimal Energy Storage Scheduling for Wind Curtailment Reduction and Energy Arbitrage: A Deep Reinforcement Learning Approach

A Control Strategy Based on Deep Reinforcement Learning Under the Combined Wind-Solar Storage System

Load Forecasting-Based Learning System for Energy Management With Battery Degradation Estimation: A Deep Reinforcement Learning Approach

Deep Reinforcement Learning-Based Energy Storage Arbitrage With Accurate Lithium-Ion Battery Degradation Model

Optimal Planning of Hybrid Energy Storage Systems using Curtailed Renewable Energy through Deep Reinforcement Learning

Stochastic dispatch of energy storage in microgrids: An augmented reinforcement learning approach

A Simulation Environment for Training a Reinforcement Learning Agent Trading a Battery Storage

Optimal energy management of energy hub: A reinforcement learning approach

Reinforcement-Learning-Based Virtual Energy Storage System Operation Strategy for Wind Power Forecast Uncertainty Management

Enhancing Battery Storage Energy Arbitrage with Deep Reinforcement Learning and Time-Series Forecasting

Operation Optimization of Wind-Energy and Storage Combined Systems: A Study on Synergizing Multifaceted Application Values of Energy Storage

Stochastic Dispatch of Energy Storage in Microgrids: A Reinforcement Learning Approach Incorporated with MCTS

Real-time Energy Management of Microgrid Using Reinforcement Learning

Reinforcement Learning for Battery Energy Storage Dispatch augmented with Model-based Optimizer