Abstract:As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.

Battery Management for Automated Warehouses via Deep Reinforcement Learning.

Battery Management for Warehouse Robots Via Average-Reward Reinforcement Learning

Real-Time Battery Thermal Management for Electric Vehicles Based on Deep Reinforcement Learning

Deep Reinforcement Learning for Dynamic Scheduling of Energy-Efficient Automated Guided Vehicles

Multi-Agent Deep Reinforcement Learning for Recharging-Considered Vehicle Scheduling Problem in Container Terminals

Deep Reinforcement Learning for Dynamic Order Picking in Warehouse Operations

Model-Free Real-Time Autonomous Energy Management for a Residential Multi-Carrier Energy System: A Deep Reinforcement Learning Approach

Enhancing Battery Storage Energy Arbitrage with Deep Reinforcement Learning and Time-Series Forecasting

Deep Reinforcement Learning for Large-Scale Inventory Management

Demand Charge Control for Energy-intensive Enterprises Based on Deep Reinforcement Learning

Autonomous Household Energy Management Using Deep Reinforcement Learning

Solving Inventory Management Problems Through Deep Reinforcement Learning

Battery Health-Aware and Deep Reinforcement Learning-Based Energy Management for Naturalistic Data-Driven Driving Scenarios

Deep Reinforcement Learning-Based Energy Storage Arbitrage With Accurate Lithium-Ion Battery Degradation Model

Can Deep Reinforcement Learning Improve Inventory Management? Performance on Dual Sourcing, Lost Sales and Multi-Echelon Problems

Handling Computation Hardness and Time Complexity Issue of Battery Energy Storage Scheduling in Microgrids by Deep Reinforcement Learning

A Deep Reinforcement Learning Approach for Inventory Control under Stochastic Lead Time and Demand

A Real-time Demand Response Strategy of Home Energy Management by Using Distributed Deep Reinforcement Learning.

Real-Time Charging Scheduling of Automated Guided Vehicles in Cyber-Physical Smart Factories Using Feature-Based Reinforcement Learning

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Supervised Reinforcement Learning for ULV Path Planning in Complex Warehouse Environment