Abstract:Considering the importance of the energy management strategy for hybrid electric vehicles , this paper is aiming at addressing the energy optimization control issue using reinforcement learning algorithms. Firstly, this paper establishes a hybrid electric vehicle power system model. Secondly, a hierarchical energy optimization control architecture based on networked information is designed, and a traffic signal timing model is used for vehicle target speed range planning in the upper system. More specifically, the optimal vehicle speed is optimized by a model predictive control algorithm. Thirdly, a mathematical model of vehicle speed variation in connected and unconnected states is established to analyze the effect of vehicle speed planning on fuel economy. Finally, three learning-based energy optimization control strategies, namely Q-learning, deep Q network (DQN), and deep deterministic policy gradient (DDPG) algorithms, are designed under the hierarchical energy optimization control architecture. It is shown that the Q-learning algorithm is able to optimize energy control; however, the agent will meet the "dimension disaster" once it faces a high-dimensional state space issue. Then, a DQN control strategy is introduced to address the problem. Due to the limitation of the discrete output of DQN, the DDPG algorithm is put forward to achieve continuous action control. In the simulation, the superiority of the DDPG algorithm over Q-learning and DQN algorithms in hybrid electric vehicles is illustrated in terms of its robustness and faster convergence for better energy management purposes.

What problem does this paper attempt to address?

The problem this paper attempts to address is the energy management control of Hybrid Electric Vehicles (HEV), specifically how to optimize energy management strategies using reinforcement learning algorithms to improve fuel economy and driving comfort. Specifically, the paper proposes a hierarchical energy optimization control architecture based on network information and designs three reinforcement learning-based energy optimization control strategies (namely Q-learning, Deep Q-Network (DQN), and Deep Deterministic Policy Gradient (DDPG) algorithms) to tackle the issues of high-dimensional state space and continuous action control. ### Main Contributions of the Paper: 1. **Hierarchical Energy Optimization Control Architecture**: - A hierarchical energy optimization control architecture is designed based on network information obtained from Vehicle-to-Infrastructure (V2I) and Vehicle-to-Vehicle (V2V) communication. - The upper-level system uses the Signal Phase and Timing (SPAT) model to plan the target speed range of the vehicle and optimizes the vehicle speed using the Model Predictive Control (MPC) algorithm. - The lower-level system performs energy optimization control based on the optimal speed planned by the upper-level system. 2. **Application of Reinforcement Learning Algorithms**: - Three reinforcement learning algorithms, Q-learning, DQN, and DDPG, are designed to address the issues of high-dimensional state space and continuous action control. - The Q-learning algorithm performs well in handling low-dimensional state space but encounters the "curse of dimensionality" in high-dimensional state space. - The DQN algorithm can handle high-dimensional state space but can only output discrete actions. - The DDPG algorithm can achieve continuous action control, has better robustness, and faster convergence speed, making it suitable for energy management of hybrid electric vehicles. ### Research Background: - With the increasing demand for energy and stricter emission standards, developing new energy vehicles has become an important measure to alleviate the global energy crisis and environmental pollution. - Hybrid Electric Vehicles (HEV) are considered one of the best solutions to environmental and energy issues due to their longer range and flexible operating modes. - Energy management control strategies are a hot topic in HEV research, with existing strategies including rule-based, optimization-based, and learning-based methods. - The application of Reinforcement Learning (RL) in HEV energy management is increasing, but existing methods have limitations in handling high-dimensional state space and continuous action control. ### Methods and Results: - **Model Establishment**: - A power system model of HEV is established, including the engine, motor, battery, and planetary gear mechanism. - Engine fuel consumption model, motor efficiency model, and battery equivalent circuit model (ECM) are established. - **Hierarchical Control Architecture**: - The upper-level system obtains vehicle position, speed, and traffic light status information through V2I and V2V communication, and uses the SPAT model to calculate the optimal target speed range. - The lower-level system uses the MPC algorithm to optimize vehicle speed and allocates power between the engine and motor through reinforcement learning algorithms to improve fuel economy. - **Simulation Results**: - Simulations were conducted under suburban and urban conditions to verify the effectiveness of the speed planning algorithm based on network information. - The results show that after using network information for speed planning, the vehicle passes through traffic intersections with green lights without stopping, improving fuel economy and driving comfort. ### Conclusion: - The hierarchical energy optimization control architecture and reinforcement learning algorithms based on network information proposed in this paper can effectively optimize the energy management of HEV, improving fuel economy and driving comfort. - In particular, the DDPG algorithm performs excellently in handling high-dimensional state space and continuous action control, with better robustness and faster convergence speed.

A deep reinforcement learning approach to energy management control with connected information for hybrid electric vehicles

Reinforcement Learning-Based Energy Management Control Strategy of Hybrid Electric Vehicles

Energy Management of Hybrid Electric Vehicles Based on Model Predictive Control and Deep Reinforcement Learning

Reinforcement Learning-Based Heuristic Planning for Optimized Energy Management in Power-Split Hybrid Electric Heavy Duty Vehicles

Deep Reinforcement Learning Based Energy Management for a Hybrid Electric Vehicle

Multi-objective Control and Energy Management Strategy Based on Deep Q-network for Parallel Hybrid Electric Vehicles

An Energy Management Strategy of Power-Split Hybrid Electric Vehicles Using Reinforcement Learning

Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle

Deep Reinforcement Learning Based Adaptive Energy Management for Plug-in Hybrid Electric Vehicle with Double Deep Q-network

Deep Reinforcement Learning-Based Energy Management for Hybrid Electric Vehicles

Continuous Reinforcement Learning of Energy Management with Deep Q Network for a Power Split Hybrid Electric Bus

Energy Management Strategy Based on Deep Reinforcement Learning and Speed Prediction for Power‐split Hybrid Electric Vehicle with Multi‐dimensional Continuous Control

Modeling and Control System Optimization for Electrified Vehicles: A Data-Driven Approach

Energy Management Strategy Based on Deep Reinforcement Learning and Speed Prediction for Power-Split Hybrid Electric Vehicle with Multidimensional Continuous Control

Energy Management Based on D4QN Reinforcement Learning for a Series-Parallel Multi-Speed Hybrid Electric Vehicle

Human-like Energy Management Based on Deep Reinforcement Learning and Historical Driving Experiences

Energy management for hybrid electric vehicles based on imitation reinforcement learning

Data-driven modeling and supervisory control system optimization for plug-in hybrid electric vehicles

A Deep Reinforcement Learning Based Hierarchical Eco-Driving Strategy for Connected and Automated HEVs

Self-supervised reinforcement learning-based energy management for a hybrid electric vehicle

Fast Learning-based Control for Energy Management of Hybrid Electric Vehicles