Abstract:Reinforcement learning (RL) has shown a promising performance in learning optimal policies for a variety of sequential decision-making tasks. However, in many real-world RL problems, besides optimizing the main objectives, the agent is expected to satisfy a certain level of safety (e.g., avoiding collisions in autonomous driving). While RL problems are commonly formalized as Markov decision processes (MDPs), safety constraints are incorporated via constrained Markov decision processes (CMDPs). Although recent advances in safe RL have enabled learning safe policies in CMDPs, these safety requirements should be satisfied during both training and in the deployment process. Furthermore, it is shown that in memory-based and partially observable environments, these methods fail to maintain safety over unseen out-of-distribution observations. To address these limitations, we propose a Lyapunov-based uncertainty-aware safe RL model. The introduced model adopts a Lyapunov function that converts trajectory-based constraints to a set of local linear constraints. Furthermore, to ensure the safety of the agent in highly uncertain environments, an uncertainty quantification method is developed that enables identifying risk-averse actions through estimating the probability of constraint violations. Moreover, a Transformers model is integrated to provide the agent with memory to process long time horizons of information via the self-attention mechanism. The proposed model is evaluated in grid-world navigation tasks where safety is defined as avoiding static and dynamic obstacles in fully and partially observable environments. The results of these experiments show a significant improvement in the performance of the agent both in achieving optimality and satisfying safety constraints.

Safe Reinforcement Learning in Autonomous Driving with Epistemic Uncertainty Estimation

Safe Reinforcement Learning Using Finite-Horizon Gradient-based Estimation

Lyapunov-based uncertainty-aware safe reinforcement learning

Uncertainty-Aware Model-Based Reinforcement Learning: Methodology and Application in Autonomous Driving

Identify, Estimate and Bound the Uncertainty of Reinforcement Learning for Autonomous Driving

Belief State Separated Reinforcement Learning for Autonomous Vehicle Decision Making under Uncertainty.

Tactical Decision-Making in Autonomous Driving by Reinforcement Learning with Uncertainty Estimation

How to Learn from Risk: Explicit Risk-Utility Reinforcement Learning for Efficient and Safe Driving Strategies

Safe-State Enhancement Method for Autonomous Driving Via Direct Hierarchical Reinforcement Learning.

Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Driving

Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints

Uncertainty-Aware Decision-Making for Autonomous Driving at Uncontrolled Intersections

Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

Safe Reinforcement Learning on Autonomous Vehicles

SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Driving

Uncertainty-Aware Model-Based Reinforcement Learning with Application to Autonomous Driving

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

Disentangling Uncertainty for Safe Social Navigation using Deep Reinforcement Learning

Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement

Model-Based Safe Reinforcement Learning With Time-Varying Constraints: Applications to Intelligent Vehicles

Safe Driving Via Expert Guided Policy Optimization