Abstract:The dynamic nature of driving environments and the presence of diverse road users pose significant challenges for decision-making in autonomous driving. Deep reinforcement learning (DRL) has emerged as a popular approach to tackle this problem. However, the application of existing DRL solutions is mainly confined to simulated environments due to safety concerns, impeding their deployment in real-world. To overcome this limitation, this paper introduces a novel neuro-symbolic model-free DRL approach, called DRL with Symbolic Logics (DRLSL) that combines the strengths of DRL (learning from experience) and symbolic first-order logics (knowledge-driven reasoning) to enable safe learning in real-time interactions of autonomous driving within real environments. This innovative approach provides a means to learn autonomous driving policies by actively engaging with the physical environment while ensuring safety. We have implemented the DRLSL framework in autonomous driving using the highD dataset and demonstrated that our method successfully avoids unsafe actions during both the training and testing phases. Furthermore, our results indicate that DRLSL achieves faster convergence during training and exhibits better generalizability to new driving scenarios compared to traditional DRL methods.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to ensure the safety of the deep reinforcement learning (DRL) system in autonomous driving, especially avoiding the occurrence of unsafe behaviors during the exploration phase**. ### Problem Background The dynamics of the autonomous driving environment and diverse road users pose significant challenges to decision - making. Although deep reinforcement learning (DRL), as a popular method, can handle these problems, existing DRL solutions are mainly limited to simulated environments because there are safety hazards in practical applications, which hinder their deployment in the real world. ### Core Objectives of the Paper To solve this problem, this paper proposes a new neural - symbolic model - free deep reinforcement learning method, called **Deep Reinforcement Learning with Symbolic Logic (DRLSL)**. This method combines the advantages of DRL (learning from experience) and symbolic first - order logic (knowledge - based reasoning), enabling the autonomous driving system to learn safely in real - time interaction with the real environment. ### Main Contributions 1. **Introduction of the DRLSL Framework**: Combine symbolic first - order logics (FOLs) with traditional DRL techniques to ensure the safety of the autonomous driving system during the DRL exploration phase. 2. **Performance Comparison**: By comparing with traditional DRL methods, the advantages of the DRLSL method are demonstrated, including ensuring safety, improving learning efficiency, and better generalization ability. ### Method Overview The DRLSL method is implemented through the following steps: - **Define the Environment Setup and Human Background Knowledge (BK)**: Use facts to represent system states, such as position, speed, acceleration, etc., and based on this information, define which actions are safe and which are unsafe. - **Filter Unsafe Actions**: Determine the set of safe actions \( A_{\text{safe}}^t \) at each time step through a symbolic logic program (SLP). - **Select a Safe Action**: The DRL agent uses the ϵ - greedy method to select a safe action \( a_{\text{safe}}^t \) from \( A_{\text{safe}}^t \). - **Execute the Action and Update the Network**: Execute the selected action, observe the reward and the next state, then calculate the loss function and update the weights of the Q - network. In this way, DRLSL ensures that the agent only selects safe actions, thereby avoiding the occurrence of unsafe behaviors and improving the reliability and safety of the autonomous driving system. ### Summary The main purpose of the paper is to ensure the safety of the autonomous driving system during the exploration phase by introducing a neural - symbolic method that combines symbolic logic and deep reinforcement learning, while also improving learning efficiency and generalization ability. This method can not only reduce the risk of accidents but also provide a transparent and interpretable safety reasoning framework for system verification and certification.

Towards Safe Autonomous Driving Policies using a Neuro-Symbolic Deep Reinforcement Learning Approach

Deep Reinforcement Learning with Enhanced Safety for Autonomous Highway Driving

Autonomous Highway Driving using Deep Reinforcement Learning

Safe-State Enhancement Method for Autonomous Driving Via Direct Hierarchical Reinforcement Learning.

From Naturalistic Traffic Data to Learning-Based Driving Policy: A Sim-to-Real Study

Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving

Safe Driving Of Autonomous Vehicles Through Improved Deep Reinforcement Learning

Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints

Reinforcement Learning Based Safe Decision Making for Highway Autonomous Driving

Exploiting Multi-Modal Fusion for Urban Autonomous Driving Using Latent Deep Reinforcement Learning

Symbolic Imitation Learning: From Black-Box to Explainable Driving Policies

A Review on Deep Reinforcement Learning for Autonomous Driving

A Survey of Deep RL and IL for Autonomous Driving Policy Learning

Deep-Reinforcement-Learning-Based Collision Avoidance of Autonomous Driving System for Vulnerable Road User Safety

How to Learn from Risk: Explicit Risk-Utility Reinforcement Learning for Efficient and Safe Driving Strategies

Comprehensive Training and Evaluation on Deep Reinforcement Learning for Automated Driving in Various Simulated Driving Maneuvers

Self-Learned Autonomous Driving at Unsignalized Intersections: A Hierarchical Reinforced Learning Approach for Feasible Decision-Making

Integrating Deep Reinforcement Learning with Model-based Path Planners for Automated Driving

Towards Robust Decision-Making for Autonomous Highway Driving Based on Safe Reinforcement Learning

Deep Reinforcement Learning on Autonomous Driving Policy With Auxiliary Critic Network

Automated Driving Maneuvers under Interactive Environment based on Deep Reinforcement Learning