Abstract:As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.

Deep Reinforcement Learning-Based Method for Joint Optimization of Mobile Energy Storage Systems and Power Grid with High Renewable Energy Sources

A Holistic Power Optimization Approach for Microgrid Control Based on Deep Reinforcement Learning

Double Deep Q-learning Based Real-Time Optimization Strategy for Microgrids

Multi-agent Deep Reinforcement Learning for Resilience-Driven Routing and Scheduling of Mobile Energy Storage Systems

Deep Reinforcement Learning Based Optimal Energy Management of Multi-energy Microgrids with Uncertainties

Multi-Objective Interval Optimization Dispatch of Microgrid Via Deep Reinforcement Learning

Physical-model-free intelligent energy management for a grid-connected hybrid wind-microturbine-PV-EV energy system via deep reinforcement learning approach

Routing and Scheduling of Mobile Energy Storage System for Electricity Arbitrage Based on Two-Layer Deep Reinforcement Learning

Deep Reinforcement Learning Based Double-layer Optimization Method for Energy Management of Microgrid

Research on Capacity Configuration Optimization of Multi-Energy Complementary System Using Deep Reinforce Learning

A Control Strategy Based on Deep Reinforcement Learning Under the Combined Wind-Solar Storage System

Multi-agent deep reinforcement learning-based optimal energy management for grid-connected multiple energy carrier microgrids

Deep Reinforcement Learning–based Approach for Optimizing Energy Conversion in Integrated Electrical and Heating System with Renewable Energy

Optimizing Load Scheduling in Power Grids Using Reinforcement Learning and Markov Decision Processes

Multi-Objective Reinforcement Learning based Multi-Microgrid System Optimisation Problem

Physical model-assisted deep reinforcement learning for energy management optimization of industrial electric-hydrogen coupling system with hybrid energy storage

Deep Reinforcement Learning-based Electric Vehicle Charging Optimization with Renewable Generations

Multi-objective two-stage robust optimization of wind/PV/thermal power system based on meta multi-agent reinforcement learning

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Deep Reinforcement Learning for Stochastic Dynamic Microgrid Energy Management

Optimal Operation Strategy for Distribution Grid with High Penetration Renewable Energy: a Novel Deep Reinforcement Learning Based Approach