Abstract:The dynamic conditions and internal states of portable energy storage system (PESS), such as temperature, electricity price, state of charge (SOC), and state of health (SOH), significantly impact battery degradation. Current decision-making models for PESS operation often oversimplify the modeling of battery degradation. To address this, we introduce an environment-adaptive online learning framework that effectively integrates deep neural networks and reinforcement learning to exploit and explore external environments (i.e., electricity prices and temperature) and internal dynamics (i.e., battery degradation), providing decision support for PESS operation. This framework dynamically updates battery degradation and decision-making models in real-time, enhancing adaptive responses to external changes. Specifically, we developed a neural network based on porous electrode theory that considers multi-physical factors, such as charging power, initial and terminal SOC, SOH, and temperature to accurately assess battery degradation. This network is embedded within a deep reinforcement learning algorithm, enabling real-time, adaptive decision-making for PESS amidst varying environmental conditions. Furthermore, to navigate complex operational environments, a fine-tuning mechanism is incorporated into the degradation neural network. Application of this framework to the energy arbitrage of PESS in the California power grid demonstrates an average benefit increase of 37% compared to traditional degradation assessment models. Note to Practitioners —In this work, we develop a novel approach to addressing the critical issue of battery degradation in PESS. Existing models often oversimplify degradation, hindering accurate assessments of performance and lifespan projections. More recently, learning-based algorithms have demonstrated outstanding performance in both battery degradation modeling and real-time decision-making. In this sense, we introduce a sophisticated neural network model grounded in a porous electrode model. This model considers multi-physics factors involving charging/discharging power, initial and terminal SOC, SOH, and temperature. Complementing this, we integrate the aforementioned model into an online learning framework, enabling real-time decision-making for PESS. Furthermore, to tackle the challenges of complex operating environments, a fine-tuning mechanism is incorporated into the battery degradation neural network. We validate the effectiveness of the proposed methods through an energy arbitrage application of PESS and also reveal its potential for on-demand applications in energy and transportation systems. This note anticipates a positive impact on PESS management and contributes significantly to the evolution of energy storage systems, offering practitioners invaluable decision support for commercial applications involving battery sharing, trading, and renting.

Deep Reinforcement Learning-Based Energy Storage Arbitrage With Accurate Lithium-Ion Battery Degradation Model

Enhancing Battery Storage Energy Arbitrage with Deep Reinforcement Learning and Time-Series Forecasting

Load Forecasting-Based Learning System for Energy Management With Battery Degradation Estimation: A Deep Reinforcement Learning Approach

Routing and Scheduling of Mobile Energy Storage System for Electricity Arbitrage Based on Two-Layer Deep Reinforcement Learning

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Arbitrage of Energy Storage in Electricity Markets with Deep Reinforcement Learning

Reinforcement Learning-Based Optimal Battery Control Under Cycle-Based Degradation Cost

Energy Storage Arbitrage in Real-Time Markets via Reinforcement Learning

Distributional Reinforcement Learning-based Energy Arbitrage Strategies in Imbalance Settlement Mechanism

Environment-Adaptive Online Learning for Portable Energy Storage Based on Porous Electrode Model

Developing Optimal Energy Arbitrage Strategy for Energy Storage System Using Reinforcement Learning

Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies

Temporal-Aware Deep Reinforcement Learning for Energy Storage Bidding in Energy and Contingency Reserve Markets

Residential Energy Arbitrage Considering Demand Response and Accurate Battery Model with Degradation Cost

Reinforcement Learning Driven Self-Optimizing Operation for Distributed Battery Electrical Storage System

Degradation Prediction and Cost Optimization of Second-Life Battery Used for Energy Arbitrage and Peak-Shaving in an Electric Grid

Risk-Sensitive Mobile Battery Energy Storage System Control With Deep Reinforcement Learning and Hybrid Risk Estimation Method

Degradation-Aware Derating of Lithium-Ion Battery Energy Storage Systems in the UK Power Market

A Simulation Environment for Training a Reinforcement Learning Agent Trading a Battery Storage

Reinforcement Learning-Enhanced Adaptive Scheduling of Battery Energy Storage Systems in Energy Markets

Improving optimal control of grid-connected lithium-ion batteries through more accurate battery and degradation modelling