Abstract:As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.

Deep Reinforcement Learning for Artificial Upwelling Energy Management

Deep reinforcement learning for artificial upwelling energy management

Layout Optimization for Underwater Nozzle Array of Air-Lifted Artificial Upwelling System Based on Discrete Particle Swarm Algorithm

An end-to-end sensor scheduling method based on D3QN for underwater passive tracking in UWSNs

A Novel Sensor Scheduling Algorithm Based on Deep Reinforcement Learning for Bearing-Only Target Tracking in UWSNs

Energy optimal dispatching of ship's integrated power system based on deep reinforcement learning

AUV-Aided Optical-Acoustic Hybrid Data Collection Based on Deep Reinforcement Learning

Deep Reinforcement Learning for Risk and Disaster Management in Energy-Efficient Marine Ranching

A Hierarchical Deep-Reinforcement Approach

Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle

An AUV-assisted Data Collection Scheme for UWSNs Based on Reinforcement Learning under the Influence of Ocean Current

Towards a Sustainable Internet-of-Underwater-Things based on AUVs, SWIPT, and Reinforcement Learning

Deep Reinforcement Learning for Channel and Power Allocation in UAV-enabled IoT Systems

A Data-Efficient Training Method for Deep Reinforcement Learning

Enhancing Information Freshness: An AoI Optimized Markov Decision Process Dedicated In the Underwater Task

Maximizing UAV Coverage in Maritime Wireless Networks: A Multiagent Reinforcement Learning Approach

A Deep Reinforcement Learning Based Approach for Optimal Active Power Dispatch

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Real-time Energy Management of a Low-Carbon Micro-Grid Based on Deep Reinforcement Learning

Data-driven stochastic energy management of multi energy system using deep reinforcement learning

Target Search Control Of Auv In Underwater Environment With Deep Reinforcement Learning