Reinforcement Learning Assisted Deep Learning for Probabilistic Charging Power Forecasting of EVCS

Yuanzheng Li,Yong Zhao,Lei Wu,Zhigang Zeng

2023-01-01

Abstract:As one of the most essential actions, the carbon emission reduction has been adopted to address the problem of climate change.

What problem does this paper attempt to address?

Probabilistic Charging Power Forecast of EVCS: Reinforcement Learning Assisted Deep Learning Approach

Yuanzheng Li,Shangyang He,Yang Li,Leijiao Ge,Suhua Lou,Zhigang Zeng

DOI: https://doi.org/10.1109/TIV.2022.3168577

2022-04-17

Abstract:The electric vehicle (EV) and electric vehicle charging station (EVCS) have been widely deployed with the development of large-scale transportation electrifications. However, since charging behaviors of EVs show large uncertainties, the forecasting of EVCS charging power is non-trivial. This paper tackles this issue by proposing a reinforcement learning assisted deep learning framework for the probabilistic EVCS charging power forecasting to capture its uncertainties. Since the EVCS charging power data are not standard time-series data like electricity load, they are first converted to the time-series format. On this basis, one of the most popular deep learning models, the long short-term memory (LSTM) is used and trained to obtain the point forecast of EVCS charging power. To further capture the forecast uncertainty, a Markov decision process (MDP) is employed to model the change of LSTM cell states, which is solved by our proposed adaptive exploration proximal policy optimization (AePPO) algorithm based on reinforcement learning. Finally, experiments are carried out on the real EVCSs charging data from Caltech, and Jet Propulsion Laboratory, USA, respectively. The results and comparative analysis verify the effectiveness and outperformance of our proposed framework.

Systems and Control
Multi-service Provision for Electric Vehicles in Power-Transportation Networks Towards a Low-Carbon Transition: A Hierarchical and Hybrid Multi-Agent Reinforcement Learning Approach

Dawei Qiu,Yi Wang,Mingyang Sun,Goran Strbac

DOI: https://doi.org/10.1016/j.apenergy.2022.118790

IF: 11.2

2022-01-01

Applied Energy

Abstract:In order to achieve the target of carbon peak and carbon neutrality, electric vehicles (EVs) have increasingly received a prominent interest to electrify the transportation sector due to their advantages of mobility and flexibility on handling complicated transportation and power networks. However, it is still challenging to realize the significant potential of EVs towards an emerging low-carbon transition. Previous works have focused on vehicle-to-grid (V2G) technology that allows for an increased utilization of EVs to make arbitrage by the temporal differentials of electricity prices. Nevertheless, the economic potential of EVs flexibility may not be fully exploited lacking an appropriate business model. This paper addresses this challenge by developing a coupled power-transportation network for cooperative EVs to optimize the provision of multiple inter-dependent services, including charging service, demand management service, carbon intensity service, and balancing service. In order to unlock this value, the EVs operation problem has already been tackled using model-based optimization approaches, which may raise privacy issues since the requirement for global information and also can be time consuming due to the high variability of transportation and power networks. In this paper, we propose a model-free hierarchical and hybrid multi-agent reinforcement learning method to learn the routing and scheduling decisions of EVs in a coupled power-transportation network with the objective of optimizing multi-service provisions. To this end, EVs do not reply on any knowledge of the simulated environment and are capable of handling system uncertainties via the learning process. Extensive case studies based on a 15-bus radial power distribution network and a 9-node 12-edge transportation network are developed to show that the proposed method outperforms the conventional learning algorithms in terms of policy quality and convergence speed. Finally, the generalizability and scalability are also investigated for different environment circumstances and EV numbers.
Hybrid Multi-Agent Reinforcement Learning for Electric Vehicle Resilience Control Towards a Low-Carbon Transition

Dawei Qiu,Yi Wang,Tingqi Zhang,Mingyang Sun,Goran Strbac

DOI: https://doi.org/10.1109/tii.2022.3166215

IF: 12.3

2022-01-01

IEEE Transactions on Industrial Informatics

Abstract:In responseto low-carbon requirements, a large amount of renewable energy resources (RESs) have been deployed in power systems; nevertheless, the intermittency of RESs raises the system vulnerability and even causes severe damage under extreme events. Electric vehicles (EVs), owing to their mobility and flexibility characteristics, can provide various ancillary services meanwhile enhancing system resilience. The distributed control of EVs under such scenarios in power-transportation network becomes a complex decision-making problem with enormous dynamics and uncertainties. To this end, a multiagent reinforcement learning method is proposed to compute both discrete and continuous actions simultaneously that aligns with the nature of EV routing and scheduling problems. Furthermore, the proposed method can enhance the learning stability and scalability with privacy perseverance in the multiagent setting. Simulation results based on IEEE 6- and 33-bus power networks integrated with transportation systems validate its effectiveness in providing system resilience and carbon intensity service.
Deep Reinforcement Learning Based Charging Scheduling for Household Electric Vehicles in Active Distribution Network

Taoyi Qi,Chengjin Ye,Yuming Zhao,Lingyang Li,Yi Ding

DOI: https://doi.org/10.35833/mpce.2022.000456

IF: 4.469

2023-01-01

Journal of Modern Power Systems and Clean Energy

Abstract:With the booming of electric vehicles (EVs) across the world, their increasing charging demands pose challenges to urban distribution networks. Particularly, due to the further implementation of time-of-use prices, the charging behaviors of household EVs are concentrated on low-cost periods, thus generating new load peaks and affecting the secure operation of the medium- and low-voltage grids. This problem is particularly acute in many old communities with relatively poor electricity infrastructure. In this paper, a novel two-stage charging scheduling scheme based on deep reinforcement learning is proposed to improve the power quality and achieve optimal charging scheduling of household EVs simultaneously in active distribution network (ADN) during valley period. In the first stage, the optimal charging profiles of charging stations are determined by solving the optimal power flow with the objective of eliminating peak-valley load differences. In the second stage, an intelligent agent based on proximal policy optimization algorithm is developed to dispatch the household EVs sequentially within the low-cost period considering their discrete nature of arrival. Through powerful approximation of neural network, the challenge of imperfect knowledge is tackled effectively during the charging scheduling process. Finally, numerical results demonstrate that the proposed scheme exhibits great improvement in relieving peak-valley differences as well as improving voltage quality in the ADN.
Optimal Electric Vehicle Charging Strategy with Markov Decision Process and Reinforcement Learning Technique

Tao Ding,Ziyu Zeng,Jiawen Bai,Boyu Qin,Yongheng Yang,Mohammad Shahidehpour

DOI: https://doi.org/10.1109/tia.2020.2990096

IF: 4.079

2020-01-01

IEEE Transactions on Industry Applications

Abstract:Electric vehicles (EVs) have rapidly developed in recent years and their penetration has also significantly increased, which, however, brings new challenges to power systems. Due to their stochastic behaviors, the improper charging strategies for EVs may violate the voltage security region. To address this problem, an optimal EV charging strategy in a distribution network is proposed to maximize the profit of the distribution system operators while satisfying all the physical constraints. When dealing with the uncertainties from EVs, a Markov decision process model is built to characterize the time series of the uncertainties, and then the deep deterministic policy gradient based reinforcement learning technique is utilized to analyze the impact of uncertainties on the charging strategy. Finally, numerical results verify the effectiveness of the proposed method.
Reinforcement Learning and Stochastic Optimization with Deep Learning-Based Forecasting on Power Grid Scheduling

Cheng Yang,Jihai Zhang,Wei Jiang,Li Wang,Hanwei Zhang,Zhongkai Yi,Fangquan Lin

DOI: https://doi.org/10.3390/pr11113188

IF: 3.5

2023-11-09

Processes

Abstract:The emission of greenhouse gases is a major contributor to global warming. Carbon emissions from the electricity industry account for over 40% of the total carbon emissions. Researchers in the field of electric power are making efforts to mitigate this situation. Operating and maintaining the power grid in an economic, low-carbon, and stable environment is challenging. To address the issue, we propose a grid dispatching technique that combines deep learning-based forecasting technology, reinforcement learning, and optimization technology. Deep learning-based forecasting can forecast future power demand and solar power generation, while reinforcement learning and optimization technology can make charging and discharging decisions for energy storage devices based on current and future grid conditions. In the optimization method, we simplify the complex electricity environment to speed up the solution. The combination of proposed deep learning-based forecasting and stochastic optimization with online data augmentation is used to address the uncertainty of the dispatch system. A multi-agent reinforcement learning method is proposed to utilize team reward among energy storage devices. At last, we achieved the best results by combining reinforcement and optimization strategies. Comprehensive experiments demonstrate the effectiveness of our proposed framework.

engineering, chemical
Electric Vehicle Charging Management Based on Deep Reinforcement Learning

Sichen Li,Weihao Hu,Di Cao,Tomislav Dragicevic,Qi Huang,Zhe Chen,Frede Blaabjerg

DOI: https://doi.org/10.35833/MPCE.2020.000460

IF: 4.469

2022-01-01

Journal of Modern Power Systems and Clean Energy

Abstract:A time-variable time-of-use electricity price can be used to reduce the charging costs for electric vehicle (EV) owners. Considering the uncertainty of price fluctuation and the randomness of EV owner's commuting behavior, we propose a deep reinforcement learning based method for the minimization of individual EV charging cost. The charging problem is first formulated as a Markov decision process (MDP), which has unknown transition probability. A modified long short-term memory (LSTM) neural network is used as the representation layer to extract temporal features from the electricity price signal. The deep deterministic policy gradient (DDPG) algorithm, which has continuous action spaces, is used to solve the MDP. The proposed method can automatically adjust the charging strategy according to electricity price to reduce the charging cost of the EV owner. Several other methods to solve the charging problem are also implemented and quantitatively compared with the proposed method which can reduce the charging cost up to 70.2% compared with other benchmark methods.
Dynamic Balancing-Charging Management for Shared Autonomous Electric Vehicle Systems: A Two-Stage Learning-Based Approach

Xiaolei Zhu,Yang Liu,Jiaohong Xie,Xindi Tang

DOI: https://doi.org/10.1109/ITSC57777.2023.10422187

2023-09-24

Abstract:The proliferation of car-sharing systems, a key component of sustainable urban mobility, has significantly contributed to the shift towards green transportation modes. However, balancing supply and demand in large-scale Shared Autonomous Electric Vehicle Services (SAEVSs) remains a critical yet challenging task, particularly considering the stochastic spatio-temporal dynamics and limited charging resources. To jointly optimize fleet management and charging decisions, we propose a two-stage learning-based approach that integrates the foresight capabilities of Deep Reinforcement Learning (DRL) techniques with the precision of optimization methods. We develop a multi-agent DRL model, acting as a “Manager” that generates prescient grid-level Balancing-Charging (BC) strategies, guiding the relocation and charging decisions. Furthermore, we customize the space-time-battery network flow model to make optimal vehicle-level decisions as a “Worker”, which follows the far-sighted guidance from the Manager to make high-quality and real-time decisions. Through extensive experiments conducted on a city-scale SAEVS, we demonstrate our proposed approach significantly outperforms benchmark models in terms of both system-level profit and service quality. Our two-stage approach also lays a methodological basis for further exploring the integration of DRL and optimization techniques, with the aim of enhancing decision-making capabilities in urban mobility systems.

Computer Science,Engineering,Environmental Science
Deep Reinforcement Learning for Charging Electric Vehicles with Uncertain Photovoltaic Power

Haiyan Liang,Zhuo Wang,Guokai Hao,Li Xiao,Zhigang Sun

DOI: https://doi.org/10.1109/cac57257.2022.10055292

2022-01-01

Abstract:Electric vehicles (EVs) are some of the major consumers of renewable energy generation in recent years. However, uncertainties associated with the photovoltaic (PV) output and EV users’ charging behaviors have imposed great challenges on PV power consumption. This study presents a real-time EV charging strategy based on deep reinforcement learning (DRL) to reduce the PV curtailment and charging costs of EV users. A mathematical model is constructed to describe the charging control process of EVs, in which the uncertainties of PV outputs and users’ charging behaviors are considered. This EV charging control problem is then formulated as a Markov Decision Process, and DRL is developed to learn the optimal real-time charging strategy in a dynamic environment. A policy search algorithm based on Proximal Policy Optimization is used to train the neural network. Moreover, a novel allocation criterion is proposed for the charging behavior of a single EV, thus ensuring that each EV is fully charged before departure. The proposed approach is tested on simulation cases, and the results verify the effectiveness in improving PV accommodation with uncertainties.
Deep Reinforcement Learning-based Electric Vehicle Charging Optimization with Renewable Generations

Yuge Duan,Zhao Liu,Xihao Wang,Xiaojun Wang,Qingkai Sun,Zhiyi Zhao

DOI: https://doi.org/10.1109/cieec58067.2023.10166611

2023-01-01

Abstract:With the increasing penetrations of electric vehicles (EV) and distributed renewable generations in power distribution systems, the peak load of EVs are overlapped with the conventional peak load of the distribution systems and creates the peak curve, which challenges the system operations. In this work, a deep reinforcement learning (DRL)-based EV charging optimization strategy is proposed. Firstly, the problem is formulated as a Markov Decision Process (MDP) for DRL algorithms. Secondly, the Deep Deterministic Policy Gradient (DDPG) algorithm is implemented to optimize the charging decisions of EVs. Lastly, distributed solar generation and location energy storage systems are utilized to further improve the economy and reduce carbon emissions. The proposed DRL-based framework is tested with models and the performance is validated with real-world data.
Multi-agent DRL-based Data-Driven Approach for PEVs Charging/discharging Scheduling in Smart Grid.

Yanni Wan,Jiahu Qin,Qichao Ma,Weiming Fu,Shi Wang

DOI: https://doi.org/10.1016/j.jfranklin.2022.01.016

IF: 4.246

2022-01-01

Journal of the Franklin Institute

Abstract:This paper studies the charging/discharging scheduling problem of plug-in electric vehicles (PEVs) in smart grid, considering the users' satisfaction with state of charge (SoC) and the degradation cost of batteries. The objective is to collectively determine the energy usage patterns of all participating PEVs so as to minimize the energy cost of all PEVs while ensuring the charging needs of PEV owners. The challenges herein are mainly in three folds: 1) the randomness of electricity price and PEVs' commuting behavior; 2) the unknown dynamics model of SoC; and 3) a large solution space, which make it challenging to directly develop a model-based optimization algorithm. To this end, we first reformulate the above energy cost minimization problem as a Markov game with unknown transition probabilities. Then a multi-agent deep reinforcement learning (DRL)-based data-driven approach is developed to solve the Markov game. Specifically, the proposed approach consists of two networks: an extreme learning machine (ELM)-based feedforward neural network (NN) for uncertainty prediction of electricity price and PEVs' commuting behavior and a Q network for optimal action-value function approximation. Finally, the comparison results with three benchmark solutions show that our proposed algorithm can not only adaptively decide the optimal charging/discharging policy by on-line learning process, but also yield a lower energy cost within an unknown market environment.
Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning

Hepeng Li,Haibo He,Zhiqiang Wan,D. Prokhorov

DOI: https://doi.org/10.1109/TSG.2018.2879572

IF: 10.275

2019-09-01

IEEE Transactions on Smart Grid

Abstract:Driven by the recent advances in electric vehicle (EV) technologies, EVs have become important for smart grid economy. When EVs participate in demand response program which has real-time pricing signals, the charging cost can be greatly reduced by taking full advantage of these pricing signals. However, it is challenging to determine an optimal charging strategy due to the existence of randomness in traffic conditions, user’s commuting behavior, and the pricing process of the utility. Conventional model-based approaches require a model of forecast on the uncertainty and optimization for the scheduling process. In this paper, we formulate this scheduling problem as a Markov Decision Process (MDP) with unknown transition probability. A model-free approach based on deep reinforcement learning is proposed to determine the optimal strategy for this problem. The proposed approach can adaptively learn the transition probability and does not require any system model information. The architecture of the proposed approach contains two networks: a representation network to extract discriminative features from the electricity prices and a Q network to approximate the optimal action-value function. Numerous experimental results demonstrate the effectiveness of the proposed approach.

Computer Science,Engineering
Coherent Hierarchical Probabilistic Forecasting of Electric Vehicle Charging Demand

Kedi Zheng,Hanwei Xu,Zeyang Long,Yi Wang,Qixin Chen

DOI: https://doi.org/10.1109/TIA.2023.3344544

2024-11-04

Abstract:The growing penetration of electric vehicles (EVs) significantly changes typical load curves in smart grids. With the development of fast charging technology, the volatility of EV charging demand is increasing, which requires additional flexibility for real-time power balance. The forecasting of EV charging demand involves probabilistic modeling of high dimensional time series dynamics across diverse electric vehicle charging stations (EVCSs). This paper studies the forecasting problem of multiple EVCS in a hierarchical probabilistic manner. For each charging station, a deep learning model based on a partial input convex neural network (PICNN) is trained to predict the day-ahead charging demand's conditional distribution, preventing the common quantile crossing problem in traditional quantile regression models. Then, differentiable convex optimization layers (DCLs) are used to reconcile the scenarios sampled from the distributions to yield coherent scenarios that satisfy the hierarchical constraint. It learns a better weight matrix for adjusting the forecasting results of different targets in a machine-learning approach compared to traditional optimization-based hierarchical reconciling methods. Numerical experiments based on real-world EV charging data are conducted to demonstrate the efficacy of the proposed method.

Systems and Control,Machine Learning
A Multi-Agent Reinforcement Learning-Based Method for Multiple Electric Vehicles Charging Scheduling

Kuan Li,Ying Zhang,Tiantian Zhang,Junyi Xiao,Jun Zhou,Chenglie Du

DOI: https://doi.org/10.1117/12.3023877

2024-01-01

Abstract:The multiple electric vehicles (EVs) charging scheduling problem is a challenging issue in the context of sustainable urban mobility. To address this complex problem, a mixed reward multi-agent deep deterministic policy gradient (MRMADDPG)- based method is proposed in this paper. The proposed MR-MADDPG provides a framework for multiple EVs to collaboratively and adaptively make charging decisions in a shared charging infrastructure. By learning and adapting from interactions with the charging environment, the MR-MADDPG empowers every EV within the fleet to make real-time charging decisions based on its local observation and eventually gets relatively low charging costs. This research contributes to the advancement of sustainable urban transportation by harnessing the capabilities of MRMADDPG, promoting efficient energy use and reducing charging cost of EV.
Probability-Based Energy Reinforced Management of Electric Vehicle Aggregation in the Electrical Grid Frequency Regulation.

Chaoyu Dong,Jianwen Sun,Feng Wu,Hongjie Jia

DOI: https://doi.org/10.1109/access.2020.3002693

IF: 3.9

2020-01-01

IEEE Access

Abstract:The model uncertainties and the heterogeneous energy states burden the effective aggregation of electric vehicles (EVs), especially coupling with the real-time frequency dynamic of the electrical grid. Integrating the advantages of deep learning and reinforcement learning, deep reinforcement learning shows its potential to relieve this challenge, where an intelligent agent fully considers the individual state of charge (SOC) difference of EV and the grid state to optimize the aggregation performance. However, existing policies of deep reinforcement learning usually provide deterministic and certain actions, and it is difficult to deal with the increasing uncertainties and randomness in modern electrical systems. In this paper, a probability-based management strategy is proposed with continuous action space based on the deep reinforcement learning, which provides fine-grained energy management and addresses the time-varying dynamics from EVs and electrical grid simultaneously. Moreover, an optimization based on the proximal policy is further introduced to clip the policy upgradation speed to enhance the training stability. The effectiveness of proposed energy management structure and policy optimization strategy are verified on various scenarios and uncertainties, which demonstrates advantageous performance in the SOC management and frequency maintenance. Besides the performance merits, the training procedure is also presented revealing the evolution reason for the proposed approach.
Improving electric vehicle charging forecasting: A hybrid deep learning approach for probabilistic predictions

Ali Jamali Jahromi,Mohammad Reza Masoudi,Mohammad Mohammadi,Shahabodin Afrasiabi

DOI: https://doi.org/10.1049/gtd2.13276

2024-10-09

IET Generation Transmission & Distribution

Abstract:Electric vehicles (EVs) have gained significant attention recently. Despite their advantages, challenges in the power grid, such as providing necessary information for optimal operation, persist. High‐precision forecasting techniques are essential to address the nonlinear and complex behavior of EV charging. A hybrid structure based on deep learning, called LSTLNet, has been proposed. LSTLNet combines convolutional neural networks (CNN), gated recurrent neural networks (GRU), attention mechanisms (AM), and automatic regression (AR) models. This combination improves the deterministic forecasting model and addresses the weaknesses of CNN and GRU. Deterministic prediction, which determines only one point of consumption charge, is prone to error. Therefore, probabilistic forecasting, represented as a probability distribution function (PDF) containing comprehensive statistical information, is preferred. A smooth band limit maximum likelihood (SBLM) estimator is used to indirectly predict the PDF from the data. Comparative results with conventional shallow and deep methods for similar time series forecasting demonstrate the superiority of the proposed method for both deterministic and probabilistic forecasting. Electric vehicles (EVs) have gained significant attention recently. Despite their advantages, challenges in the power grid, such as providing necessary information for optimal operation, persist. High‐precision forecasting techniques are essential to address the nonlinear and complex behavior of EV charging. A hybrid structure based on deep learning, called LSTLNet, has been proposed. LSTLNet combines convolutional neural networks (CNN), gated recurrent neural networks (GRU), attention mechanisms (AM), and automatic regression (AR) models. This combination improves the deterministic forecasting model and addresses the weaknesses of CNN and GRU. Deterministic prediction, which determines only one point of consumption charge, is prone to error. Therefore, probabilistic forecasting, represented as a probability distribution function (PDF) containing comprehensive statistical information, is preferred. A smooth band limit maximum likelihood (SBLM) estimator is used to indirectly predict the PDF from the data. Comparative results with conventional shallow and deep methods for similar time series forecasting demonstrate the superiority of the proposed method for both deterministic and probabilistic forecasting.

engineering, electrical & electronic
Charging Load Forecasting of Electric Vehicle Based on Monte Carlo and Deep Learning

Qiang Gao,Tao Zhu,Weijiang Zhou,Guanhua Wang,Tianhan Zhang,Zhi Zhang,Muhammad Waseem,Shengyuan Liu,Chang Han,Zhenzhi Lin

DOI: https://doi.org/10.1109/ispec48194.2019.8975364

2019-01-01

Abstract:An intelligent charging load forecasting method of the electric vehicle plays a vital role in power generation and power market optimization. In this paper, electric vehicles have been categorized as electric buses, electric taxis, electric private cars and electric buses. The probabilistic models of load influencing factors are investigated, and then the calculation models of charging power for different types of electric vehicles have been attained. According to the predicted results of electric vehicle ownership, the charging load of the electric vehicle is calculated by use of Monte Carlo Simulation method for extraction of both electric vehicle initial charging state and charging time. Finally, a deep learning algorithm LSTM (Long Short Term Memory) has been utilized to predict the charging load of electric vehicles. The effectiveness of the proposed method is illustrated by the simulation results.
CDDPG: A Deep-Reinforcement-Learning-Based Approach for Electric Vehicle Charging Control

Feiye Zhang,Qingyu Yang,Dou An

DOI: https://doi.org/10.1109/JIOT.2020.3015204

IF: 10.6

2021-01-01

IEEE Internet of Things Journal

Abstract:Electric vehicle (EV) has become one of the most critical components in the smart grid with the applications of the Internet-of-Things (IoT) technologies. Real-time charging control is pivotal to ensure the efficient operation of EVs. However, the charging control performance is limited by the uncertainty of the environment. On the other hand, it is challenging to determine a charging control strategy that is able to optimize multiple objectives simultaneously. In this article, we formulate the EV charging control model as a Markov decision process (MDP) by constructing state, action, transition function, and reward. Then, we propose a deep-reinforcement-learning-based approach: charging control deep deterministic policy gradient (CDDPG) to learn the optimal charging control strategy for satisfying the user's requirement of battery energy while minimizing the user's charging expense. We utilize the long short-term memory (LSTM) network that extracts the information of previous energy price to determine the current charging control strategy. Moreover, Gaussian noise is added to the output of the actor network to prevent the agent from sticking into the nonoptimal strategy. In addition, we address the limitation of sparse rewards by using two replay buffers, of which one is used to store the rewards during the charging phase and another is used to store the rewards after charging is completed. The simulation results prove that the CDDPG-based approach outperforms the deep-Q-learning-based approach (DQL) and the deep-deterministic-policy-gradient-based approach (DDPG) in satisfying the user's requirement for the battery energy and reducing the charging cost.
Reinforcement Learning for EV Fleet Smart Charging with On-Site Renewable Energy Sources

Handong Li,Xuewu Dai,Stephen Goldrick,Richard Kotter,Nauman Aslam,Saleh Ali

DOI: https://doi.org/10.3390/en17215442

IF: 3.2

2024-10-31

Energies

Abstract:In 2020, the transportation sector was the second largest source of carbon emissions in the UK and in Newcastle upon Tyne, responsible for about 33% of total emissions. To support the UK's target of reaching net zero emissions by 2050, electric vehicles (EVs) are pivotal in advancing carbon-neutral road transportation. Optimal EV charging requires a better understanding of the unpredictable output from on-site renewable energy sources (ORES). This paper proposes an integrated EV fleet charging schedule with a proximal policy optimization method based on a framework for deep reinforcement learning. For the design of the reinforcement learning environment, mathematical models of wind and solar power generation are created. In addition, the multivariate Gaussian distributions derived from historical weather and EV fleet charging data are utilized to simulate weather and charging demand uncertainty in order to create large datasets for training the model. The optimization problem is expressed as a Markov decision process (MDP) with operational constraints. For training artificial neural networks (ANNs) through successive transition simulations, a proximal policy optimization (PPO) approach is devised. The optimization approach is deployed and evaluated on a real-world scenario comprised of council EV fleet charging data from Leicester, UK. The results show that due to the design of the rewards function and system limitations, the charging action is biased towards the time of day when renewable energy output is maximum (midday). The charging decision by reinforcement learning improves the utilization of renewable energy by 2–4% compared to the random charging policy and the priority charging policy. This study contributes to the reduction in battery charging and discharging, electricity sold to the grid to create benefits and the reduction in carbon emissions.

energy & fuels
Deep Reinforcement Learning Strategy for Electric Vehicle Charging Considering Wind Power Fluctuation

Anyun Yang,Hongbin Sun,Xiao Zhang

DOI: https://doi.org/10.25103/jestr.143.12

2021-01-01

Journal of Engineering Science and Technology Review

Abstract:Electric vehicles (EVs) can inhibit the wind power fluctuations in the generalized form of energy storage.However, optimizing the charging process of EVs under wind power fluctuations is difficult because of the uncertainties of wind power output and user demands.A charging control strategy based on deep reinforcement learning (DRL) was proposed in this study to address the influence brought by uncertain environmental factors to the control.This strategy mined the deep relation between perceiving the uncertainties of environmental factors and learning charging laws by virtue of the perceptual and learning abilities of DRL.An immediate reward mechanism that acts upon the environment was constructed from the angle of neural network fitting function.The EV charging control model was expressed as a Markov decision process (MDP) that contain the state, action, and transfer functions and reward and discount factors through temporal discretization.Next, the single-step updating and experience replay mode were combined to construct the DRL algorithm, followed by the comparative convergence experiment with the reinforcement learning (RL) algorithm that expressed the reward function in mathematical form.In the end, the agent obtained through training was used for the verification of the calculated example.Results show that the constructed RL algorithm is converged by 8,500 episodes earlier.The charging control strategy based on DRL meets the charging requirements when the proportion of optimization objectives is 0.5 and 0.9, and users are allowed to change the allowed charging time temporarily.This study demonstrates that the charging control strategy based DRL can optimize the EVs charging process under many uncertain factors.

Reinforcement Learning Assisted Deep Learning for Probabilistic Charging Power Forecasting of EVCS

Probabilistic Charging Power Forecast of EVCS: Reinforcement Learning Assisted Deep Learning Approach

Multi-service Provision for Electric Vehicles in Power-Transportation Networks Towards a Low-Carbon Transition: A Hierarchical and Hybrid Multi-Agent Reinforcement Learning Approach

Hybrid Multi-Agent Reinforcement Learning for Electric Vehicle Resilience Control Towards a Low-Carbon Transition

Deep Reinforcement Learning Based Charging Scheduling for Household Electric Vehicles in Active Distribution Network

Optimal Electric Vehicle Charging Strategy with Markov Decision Process and Reinforcement Learning Technique

Reinforcement Learning and Stochastic Optimization with Deep Learning-Based Forecasting on Power Grid Scheduling

Electric Vehicle Charging Management Based on Deep Reinforcement Learning

Dynamic Balancing-Charging Management for Shared Autonomous Electric Vehicle Systems: A Two-Stage Learning-Based Approach

Deep Reinforcement Learning for Charging Electric Vehicles with Uncertain Photovoltaic Power

Deep Reinforcement Learning-based Electric Vehicle Charging Optimization with Renewable Generations

Multi-agent DRL-based Data-Driven Approach for PEVs Charging/discharging Scheduling in Smart Grid.

Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning

Coherent Hierarchical Probabilistic Forecasting of Electric Vehicle Charging Demand

A Multi-Agent Reinforcement Learning-Based Method for Multiple Electric Vehicles Charging Scheduling

Probability-Based Energy Reinforced Management of Electric Vehicle Aggregation in the Electrical Grid Frequency Regulation.

Improving electric vehicle charging forecasting: A hybrid deep learning approach for probabilistic predictions

Charging Load Forecasting of Electric Vehicle Based on Monte Carlo and Deep Learning

CDDPG: A Deep-Reinforcement-Learning-Based Approach for Electric Vehicle Charging Control

Reinforcement Learning for EV Fleet Smart Charging with On-Site Renewable Energy Sources

Deep Reinforcement Learning Strategy for Electric Vehicle Charging Considering Wind Power Fluctuation