Abstract:Reinforcement learning (RL) has revolutionized decision-making across a wide range of domains over the past few decades. Yet, deploying RL policies in real-world scenarios presents the crucial challenge of ensuring safety. Traditional safe RL approaches have predominantly focused on incorporating predefined safety constraints into the policy learning process. However, this reliance on predefined safety constraints poses limitations in dynamic and unpredictable real-world settings where such constraints may not be available or sufficiently adaptable. Bridging this gap, we propose a novel approach that concurrently learns a safe RL control policy and identifies the unknown safety constraint parameters of a given environment. Initializing with a parametric signal temporal logic (pSTL) safety specification and a small initial labeled dataset, we frame the problem as a bilevel optimization task, intricately integrating constrained policy optimization, using a Lagrangian-variant of the twin delayed deep deterministic policy gradient (TD3) algorithm, with Bayesian optimization for optimizing parameters for the given pSTL safety specification. Through experimentation in comprehensive case studies, we validate the efficacy of this approach across varying forms of environmental constraints, consistently yielding safe RL policies with high returns. Furthermore, our findings indicate successful learning of STL safety constraint parameters, exhibiting a high degree of conformity with true environmental safety constraints. The performance of our model closely mirrors that of an ideal scenario that possesses complete prior knowledge of safety constraints, demonstrating its proficiency in accurately identifying environmental safety constraints and learning safe policies that adhere to those constraints.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is in Reinforcement Learning (RL), in the case of unknown environmental safety constraints, how to simultaneously learn a safe control policy and identify the unknown safety - constraint parameters in the environment. Traditional safe reinforcement learning methods mainly rely on predefined safety constraints, but in dynamic and unpredictable real - world environments, these predefined constraints may not be available or may be insufficiently adaptable. Therefore, the paper proposes a new method. By using parametric Signal Temporal Logic (pSTL) safety specifications and a small initial labeled data set, the problem is framed as a bilevel optimization task, combining Constrained Policy Optimization and Bayesian Optimization (BO) to optimize the parameters of the given pSTL safety specifications and generate the optimal policy that complies with the safety specifications during the learning process. This method aims to improve the safety of RL in complex dynamic environments while maintaining a high rate of return.

Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning

Joint Learning of Policy with Unknown Temporal Constraints for Safe Reinforcement Learning

Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints

Learning to be Safe: Deep RL with a Safety Critic

Probabilistic Constraint for Safety-Critical Reinforcement Learning

Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

Joint Synthesis of Safety Certificate and Safe Control Policy Using Constrained Reinforcement Learning

Model-Based Safe Reinforcement Learning With Time-Varying Constraints: Applications to Intelligent Vehicles

Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificate

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

Safe reinforcement learning for probabilistic reachability and safety specifications: A Lyapunov-based approach

Safe Reinforcement Learning for Signal Temporal Logic Tasks Using Robust Control Barrier Functions

Model-Free Safe Reinforcement Learning Through Neural Barrier Certificate

Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Driving

Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm

Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards

Learn With Imagination: Safe Set Guided State-wise Constrained Policy Optimization

Lyapunov-based uncertainty-aware safe reinforcement learning

Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints

Signal Temporal Logic Neural Predictive Control

Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning