Abstract:As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.

Deep Learning Based Online Power Control for Large Energy Harvesting Networks

Distributed Power Control for Large Energy Harvesting Networks: A Multi-Agent Deep Reinforcement Learning Approach

Deep Deterministic Policy Gradient Based Dynamic Power Control for Self-Powered Ultra-Dense Networks

Deep Reinforcement Learning-Assisted Energy Harvesting Wireless Networks

Deep Reinforcement Learning Based Approach for Optimal Power Flow of Distribution Networks Embedded with Renewable Energy and Storage Devices

Deep Reinforcement Learning for Multi-User RF Charging with Non-linear Energy Harvesters

Power Control in Energy Harvesting Multiple Access System with Reinforcement Learning.

Deep Learning-Based Distributed Optimal Control for Wide Area Energy Internet

On-line Building Energy Optimization using Deep Reinforcement Learning

A Deep Reinforcement Learning Based Approach for Optimal Active Power Dispatch

Deep Reinforcement Learning for Energy-Efficient Power Control in Heterogeneous Networks

Deep Reinforcement Learning Based Approach for Dynamic Optimal Power Flow in Active Distribution Network

Distributed Deep Reinforcement Learning for Renewable Energy Accommodation Assessment With Communication Uncertainty in Internet of Energy

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Deep Reinforcement Learning for Multi-Agent Power Control in Heterogeneous Networks

Optimal Energy Management Strategies for Energy Internet Via Deep Reinforcement Learning Approach

Deep Q-Learning Based Dynamic Resource Allocation for Self-Powered Ultra-Dense Networks

Deep Reinforcement Learning for Economic Dispatch of Virtual Power Plant in Internet of Energy.

A Data-driven Method for Fast AC Optimal Power Flow Solutions via Deep Reinforcement Learning

Maximizing Energy-Efficiency in Wireless Communication Systems Based on Deep Learning

Deep Reinforcement Learning Empowers Wireless Powered Mobile Edge Computing: Towards Energy-Aware Online Offloading