Abstract:Reinforcement learning (RL) has revolutionized decision-making across a wide range of domains over the past few decades. Yet, deploying RL policies in real-world scenarios presents the crucial challenge of ensuring safety. Traditional safe RL approaches have predominantly focused on incorporating predefined safety constraints into the policy learning process. However, this reliance on predefined safety constraints poses limitations in dynamic and unpredictable real-world settings where such constraints may not be available or sufficiently adaptable. Bridging this gap, we propose a novel approach that concurrently learns a safe RL control policy and identifies the unknown safety constraint parameters of a given environment. Initializing with a parametric signal temporal logic (pSTL) safety specification and a small initial labeled dataset, we frame the problem as a bilevel optimization task, intricately integrating constrained policy optimization, using a Lagrangian-variant of the twin delayed deep deterministic policy gradient (TD3) algorithm, with Bayesian optimization for optimizing parameters for the given pSTL safety specification. Through experimentation in comprehensive case studies, we validate the efficacy of this approach across varying forms of environmental constraints, consistently yielding safe RL policies with high returns. Furthermore, our findings indicate successful learning of STL safety constraint parameters, exhibiting a high degree of conformity with true environmental safety constraints. The performance of our model closely mirrors that of an ideal scenario that possesses complete prior knowledge of safety constraints, demonstrating its proficiency in accurately identifying environmental safety constraints and learning safe policies that adhere to those constraints.

Constrained reinforcement learning with statewise projection: a control barrier function approach

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments

Safe Reinforcement Learning via Projection on a Safe Set: How to Achieve Optimality?

Safe Reinforcement Learning Using Robust Control Barrier Functions

Model-Free Safe Reinforcement Learning Through Neural Barrier Certificate

Probabilistic Constraint for Safety-Critical Reinforcement Learning

State-Wise Safe Reinforcement Learning With Pixel Observations

End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks

Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions

State-wise Safe Reinforcement Learning: A Survey

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

Reachability Constrained Reinforcement Learning.

Safe reinforcement learning for probabilistic reachability and safety specifications: A Lyapunov-based approach

Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning

Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints

A Survey of Constraint Formulations in Safe Reinforcement Learning

Feasibility Consistent Representation Learning for Safe Reinforcement Learning

Safe Reinforcement Learning for Dynamical Systems Using Barrier Certificates

Model-based Constrained Reinforcement Learning using Generalized Control Barrier Function

Safe Reinforcement Learning on the Constraint Manifold: Theory and Applications