Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Driving

Xuemin Hu,Pan Chen,Yijun Wen,Bo Tang,Long Chen

2024-09-12

Abstract:Reinforcement learning (RL) has been widely used in decision-making and control tasks, but the risk is very high for the agent in the training process due to the requirements of interaction with the environment, which seriously limits its industrial applications such as autonomous driving systems. Safe RL methods are developed to handle this issue by constraining the expected safety violation costs as a training objective, but the occurring probability of an unsafe state is still high, which is unacceptable in autonomous driving tasks. Moreover, these methods are difficult to achieve a balance between the cost and return expectations, which leads to learning performance degradation for the algorithms. In this paper, we propose a novel algorithm based on the long and short-term constraints (LSTC) for safe RL. The short-term constraint aims to enhance the short-term state safety that the vehicle explores, while the long-term constraint enhances the overall safety of the vehicle throughout the decision-making process, both of which are jointly used to enhance the vehicle safety in the training process. In addition, we develop a safe RL method with dual-constraint optimization based on the Lagrange multiplier to optimize the training process for end-to-end autonomous driving. Comprehensive experiments were conducted on the MetaDrive simulator. Experimental results demonstrate that the proposed method achieves higher safety in continuous state and action tasks, and exhibits higher exploration performance in long-distance decision-making tasks compared with state-of-the-art methods.

Machine Learning,Artificial Intelligence,Robotics

What problem does this paper attempt to address?

The paper attempts to address the issue of safety hazards in the training process of Reinforcement Learning (RL) in autonomous driving systems. Specifically:

1. **Insufficient safety of existing reinforcement learning methods**: Traditional RL methods require extensive interaction with the environment during training, which can lead to significant safety risks, especially in the application of autonomous driving systems.

2. **Insufficient short-term state safety**: Existing safe reinforcement learning methods mainly focus on long-term safety constraints, i.e., expected cost (risk assessment) below a predetermined threshold, but fail to adequately ensure short-term state safety during the training process.

3. **The dilemma of balancing cost and reward**: These methods find it difficult to strike a balance between cost constraints and reward expectations, leading to a decline in algorithm performance.

To address the above issues, this paper proposes a safe reinforcement learning algorithm based on Long and Short-Term Constraints (LSTC). This method aims to enhance vehicle safety during the training process by introducing both short-term and long-term constraints and optimizing the training process through a dual constraint optimization method (based on Lagrange multipliers). Experimental results show that this method achieves higher safety in continuous state and action tasks and demonstrates higher exploration performance in long-distance decision-making tasks.