Abstract:Energy management strategy (EMS) is a key technology for plug-in hybrid electric vehicles (PHEVs). The energy management of certain series-parallel PHEVs involves the control of continuous variables, such as engine torque, and discrete variables, such as clutch engagement/disengagement. We establish a control-oriented model for a series-parallel plug-in hybrid system with clutch engagement control from the perspective of mixed-integer programming. Subsequently, we design an EMS based on continuous-discrete reinforcement learning (CDRL), which enables simultaneous output of continuous and discrete variables. During training, we introduce state-of-charge (SOC) randomization to ensure that the hybrid system exhibits optimal energy-saving performance in both high and low SOC. Finally, the effectiveness of the proposed CDRL strategy is verified by comparing EMS based on charge-depleting charge-sustaining (CD-CS) with rule-based clutch engagement control, and Dynamic Programming (DP). The simulation results show that, under a high SOC, the CDRL strategy proposed in this paper can improve energy efficiency by 8.3% compared to CD-CS, and the energy consumption is just 6.6% higher than the global optimum based on DP, while under a low SOC, the numbers are 4.1% and 3.9%, respectively.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to address the issue of energy management strategy (EMS) for plug-in hybrid electric vehicles (PHEV), particularly in series-parallel PHEVs, by simultaneously controlling continuous variables (such as engine torque) and discrete variables (such as clutch engagement/disengagement). Specifically, the paper focuses on optimizing energy management and clutch control through a continuous-discrete reinforcement learning (CDRL) approach to improve vehicle energy efficiency. ### Background and Motivation 1. **Importance of Energy Management**: - Energy management strategy (EMS) is a key technology for PHEVs, directly affecting the vehicle's fuel economy and emissions. - A reasonable and effective EMS is crucial for improving the overall performance of PHEVs and achieving the sustainable development goals of the automotive industry. 2. **Limitations of Existing Research**: - Rule-based, optimization-based, and learning-based methods each have their pros and cons. Rule-based methods are simple and practical but difficult to achieve a global optimal solution; optimization-based methods require prior knowledge of the driving cycle, making them hard to apply in real-world driving; existing reinforcement learning methods mainly focus on either continuous or discrete action spaces, making it difficult to handle systems with both continuous and discrete decision variables. 3. **Specific Issues**: - In series-parallel PHEVs, clutch engagement/disengagement control is a key issue. The optimal EMS needs to output both continuous variables (such as engine torque) and the discrete state of the clutch. - Existing methods face challenges such as high computational complexity and difficulty in real-time response when dealing with such mixed-integer programming problems. ### Main Contributions 1. **Modeling**: - A model of a hybrid system from the perspective of mixed-integer programming was established, treating clutch engagement/disengagement as a discrete control variable to achieve synchronous optimization of EMS and clutch engagement. 2. **Algorithm Design**: - A reinforcement learning algorithm combining a parameterized deep Q-network with twin delayed DDPG (PDQN-TD3) was designed, capable of selecting both continuous and discrete actions. 3. **SOC Randomization**: - SOC randomization was introduced during training, enabling the EMS to achieve near-optimal control under both high SOC and low SOC conditions, thus performing well in both charge-depleting and charge-sustaining modes. ### Experimental Results - Simulation results show that under high SOC, the proposed CDRL strategy improves energy efficiency by 8.3% compared to the CD-CS strategy and is only 6.6% higher than the global optimal solution based on dynamic programming (DP). - Under low SOC, the corresponding figures are 4.1% and 3.9%, respectively. ### Conclusion This paper proposes a continuous-discrete reinforcement learning method based on PDQN-TD3, effectively solving the energy management and clutch control issues in series-parallel PHEVs, significantly improving vehicle energy efficiency.

Plug-in Hybrid Electric Vehicle Energy Management with Clutch Engagement Control via Continuous-Discrete Reinforcement Learning

Plug‐In Hybrid Electric Vehicle Energy Management with Clutch Engagement Control via Continuous‐Discrete Reinforcement Learning

Energy Management for a DM-i Plug-in Hybrid Electric Vehicle via Continuous-Discrete Reinforcement Learning

An energy management strategy for plug-in hybrid electric vehicles based on deep learning and improved model predictive control

Energy Management Strategy Based on Deep Reinforcement Learning and Speed Prediction for Power-Split Hybrid Electric Vehicle with Multidimensional Continuous Control

Energy management strategy for power-split plug-in hybrid electric vehicle based on MPC and double Q-learning

The application of machine learning-based energy management strategy in a multi-mode plug-in hybrid electric vehicle, part II: Deep deterministic policy gradient algorithm design for electric mode

Transfer Deep Reinforcement Learning-Based Energy Management Strategy for Plug-In Hybrid Electric Heavy-Duty Trucks under Segmented Usage Scenarios

Battery life constrained real-time energy management strategy for hybrid electric vehicles based on reinforcement learning

An adaptive hierarchical energy management strategy for hybrid electric vehicles combining heuristic domain knowledge and data-driven deep reinforcement learning

A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning

Data-driven modeling and supervisory control system optimization for plug-in hybrid electric vehicles

Human-like Energy Management Based on Deep Reinforcement Learning and Historical Driving Experiences

An Energy Management Strategy of Power-Split Hybrid Electric Vehicles Using Reinforcement Learning

Co-Optimization of Adaptive Cruise Control and Hybrid Electric Vehicle Energy Management via Model Predictive Mixed Integer Control

Deep Reinforcement Learning based Energy Management for Heavy Duty HEV considering Discrete-Continuous Hybrid Action Space

Energy Management for Hybrid Electric Vehicles Using Safe Hybrid-Action Reinforcement Learning

Real-Time Energy Management for Plug-in Hybrid Electric Vehicles via Incorporating Double-Delay Q-Learning and Model Prediction Control

Empirical Analysis of AI-based Energy Management in Electric Vehicles: A Case Study on Reinforcement Learning

Bi-level energy management strategy for power-split plug-in hybrid electric vehicles: A reinforcement learning approach for prediction and control

A GA-based online real-time optimized energy management strategy for plug-in hybrid electric vehicles