Abstract:As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.

Duck Curve Aware Dynamic Pricing and Battery Scheduling Strategy Using Reinforcement Learning

Deep Reinforcement Learning for Strategic Bidding in Electricity Markets

Deep Reinforcement Learning Based Pricing Strategy of Aggregators Considering Renewable Energy

Price-Based Residential Demand Response Management in Smart Grids: A Reinforcement Learning-Based Approach

An Integrated Demand Response-Based Energy Management Strategy for Integrated Energy System Based on Deep Reinforcement Learning

An application of reinforcement learning to residential energy storage under real-time pricing

Optimal demand response based dynamic pricing strategy via Multi-Agent Federated Twin Delayed Deep Deterministic policy gradient algorithm

A Deep Reinforcement Learning Based Approach for Optimal Active Power Dispatch

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Dynamic Demand-Aware Power Grid Intelligent Pricing Algorithm Based on Deep Reinforcement Learning

A Control Strategy Based on Deep Reinforcement Learning Under the Combined Wind-Solar Storage System

Deep reinforcement learning-based optimal bidding strategy for real-time multi-participant electricity market with short-term load

A distributed real-time pricing strategy based on reinforcement learning approach for smart grid

Comparative Analysis of Reinforcement Learning Approaches for Multi-Objective Optimization in Residential Hybrid Energy Systems

Application-oriented assessment of grid-connected PV-battery system with deep reinforcement learning in buildings considering electricity price dynamics

Bi-level stochastic real-time pricing model in multi-energy generation system: A reinforcement learning approach

Deep Reinforcement Learning for Community Battery Scheduling under Uncertainties of Load, PV Generation, and Energy Prices

Dynamic Incentive Pricing on Charging Stations for Real-Time Congestion Management in Distribution Network: An Adaptive Model-Based Safe Deep Reinforcement Learning Method

Reward Shaping-Based Actor–Critic Deep Reinforcement Learning for Residential Energy Management

Real-time Economic Dispatch of Thermal-Wind-Battery Hybrid Systems Based on Deep Reinforcement Learning

Temporal-Aware Deep Reinforcement Learning for Energy Storage Bidding in Energy and Contingency Reserve Markets