Abstract:Reinforcement learning (RL) has shown a promising performance in learning optimal policies for a variety of sequential decision-making tasks. However, in many real-world RL problems, besides optimizing the main objectives, the agent is expected to satisfy a certain level of safety (e.g., avoiding collisions in autonomous driving). While RL problems are commonly formalized as Markov decision processes (MDPs), safety constraints are incorporated via constrained Markov decision processes (CMDPs). Although recent advances in safe RL have enabled learning safe policies in CMDPs, these safety requirements should be satisfied during both training and in the deployment process. Furthermore, it is shown that in memory-based and partially observable environments, these methods fail to maintain safety over unseen out-of-distribution observations. To address these limitations, we propose a Lyapunov-based uncertainty-aware safe RL model. The introduced model adopts a Lyapunov function that converts trajectory-based constraints to a set of local linear constraints. Furthermore, to ensure the safety of the agent in highly uncertain environments, an uncertainty quantification method is developed that enables identifying risk-averse actions through estimating the probability of constraint violations. Moreover, a Transformers model is integrated to provide the agent with memory to process long time horizons of information via the self-attention mechanism. The proposed model is evaluated in grid-world navigation tasks where safety is defined as avoiding static and dynamic obstacles in fully and partially observable environments. The results of these experiments show a significant improvement in the performance of the agent both in achieving optimality and satisfying safety constraints.

Uncertainty-Aware Model-Based Reinforcement Learning: Methodology and Application in Autonomous Driving

Uncertainty-Aware Model-Based Reinforcement Learning with Application to Autonomous Driving

Identify, Estimate and Bound the Uncertainty of Reinforcement Learning for Autonomous Driving

Uncertainty Sequence Modeling Approach for Safe and Effective Autonomous Driving

Uncertainty-Aware Decision-Making for Autonomous Driving at Uncontrolled Intersections

Belief State Separated Reinforcement Learning for Autonomous Vehicle Decision Making under Uncertainty.

Uncertainty-Aware Model-Based Offline Reinforcement Learning for Automated Driving

Safe Reinforcement Learning in Autonomous Driving with Epistemic Uncertainty Estimation

Tactical Decision-Making in Autonomous Driving by Reinforcement Learning with Uncertainty Estimation

Uncertainty-Aware Prediction and Application in Planning for Autonomous Driving: Definitions, Methods, and Comparison

Uncertainty-aware human-like driving policy learning with deep Bayesian inverse reinforcement learning

Lyapunov-based uncertainty-aware safe reinforcement learning

Prioritized Experience-Based Reinforcement Learning With Human Guidance for Autonomous Driving

Model-free Deep Reinforcement Learning for Urban Autonomous Driving

Emergency Collision Avoidance Decision-making for Autonomous Vehicles: A Model-based Reinforcement Learning Approach

Learning Resilient Behaviors for Navigation Under Uncertainty

Adaptive Stochastic Nonlinear Model Predictive Control with Look-ahead Deep Reinforcement Learning for Autonomous Vehicle Motion Control

Model-Reference Reinforcement Learning Control of Autonomous Surface Vehicles with Uncertainties

Adaptive Decision Making at the Intersection for Autonomous Vehicles Based on Skill Discovery

Continuous Decision-Making in Lane Changing and Overtaking Maneuvers for Unmanned Vehicles: A Risk-Aware Reinforcement Learning Approach with Task Decomposition

Uncertainty-aware hybrid paradigm of nonlinear MPC and model-based RL for offroad navigation: Exploration of transformers in the predictive model