Abstract:As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.

Power Allocation Strategy for Urban Rail HESS Based on Deep Reinforcement Learning Sequential Decision Optimization

Power Dynamic Allocation Strategy for Urban Rail Hybrid Energy Storage System Based on Iterative Learning Control

Deep-Reinforcement-Learning-Based Energy Management Strategy for Supercapacitor Energy Storage Systems in Urban Rail Transit

Dynamic Balancing-Charging Management for Shared Autonomous Electric Vehicle Systems: A Two-Stage Learning-Based Approach

Hybrid energy management strategy based on dynamic setting and coordinated control for urban rail train with PMSM

Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle

Reinforcement Learning Energy Management for Hybrid Electric Tracked Vehicle with Deep Deterministic Policy Gradient

A Transferable Energy Management Strategy for Hybrid Electric Vehicles Via Dueling Deep Deterministic Policy Gradient

Safe Deep Reinforcement Learning-Based Constrained Optimal Control Scheme for HEV Energy Management

Deep Reinforcement Learning based Energy Management for Heavy Duty HEV considering Discrete-Continuous Hybrid Action Space

An Intelligent Energy Management Strategy for Hybrid Vehicle with Irrational Actions Using Twin Delayed Deep Deterministic Policy Gradient

Data-driven transferred energy management strategy for hybrid electric vehicles via deep reinforcement learning

Rule-interposing Deep Reinforcement Learning Based Energy Management Strategy for Power-Split Hybrid Electric Vehicle

Distributed Deep Reinforcement Learning-Based Energy and Emission Management Strategy for Hybrid Electric Vehicles.

Reinforcement Learning–Based Energy Management Strategy for a Hybrid Electric Tracked Vehicle

Continuous Reinforcement Learning of Energy Management with Deep Q Network for a Power Split Hybrid Electric Bus

Energy Management Optimization for Connected Hybrid Electric Vehicle Using Offline Reinforcement Learning

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

An adaptive hierarchical energy management strategy for hybrid electric vehicles combining heuristic domain knowledge and data-driven deep reinforcement learning

Deep Reinforcement Learning-Based Spatiotemporal Decision of Utility-Scale Highway Portable Energy Storage Systems

Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle