Abstract:Reinforcement learning (RL) has revolutionized decision-making across a wide range of domains over the past few decades. Yet, deploying RL policies in real-world scenarios presents the crucial challenge of ensuring safety. Traditional safe RL approaches have predominantly focused on incorporating predefined safety constraints into the policy learning process. However, this reliance on predefined safety constraints poses limitations in dynamic and unpredictable real-world settings where such constraints may not be available or sufficiently adaptable. Bridging this gap, we propose a novel approach that concurrently learns a safe RL control policy and identifies the unknown safety constraint parameters of a given environment. Initializing with a parametric signal temporal logic (pSTL) safety specification and a small initial labeled dataset, we frame the problem as a bilevel optimization task, intricately integrating constrained policy optimization, using a Lagrangian-variant of the twin delayed deep deterministic policy gradient (TD3) algorithm, with Bayesian optimization for optimizing parameters for the given pSTL safety specification. Through experimentation in comprehensive case studies, we validate the efficacy of this approach across varying forms of environmental constraints, consistently yielding safe RL policies with high returns. Furthermore, our findings indicate successful learning of STL safety constraint parameters, exhibiting a high degree of conformity with true environmental safety constraints. The performance of our model closely mirrors that of an ideal scenario that possesses complete prior knowledge of safety constraints, demonstrating its proficiency in accurately identifying environmental safety constraints and learning safe policies that adhere to those constraints.

Lagrangian-based online safe reinforcement learning for state-constrained systems

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

Safe Intermittent Reinforcement Learning for Nonlinear Systems.

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

Robust Safe Reinforcement Learning Control of Unknown Continuous-Time Nonlinear Systems with State Constraints and Disturbances

End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks

Sample-efficient Safe Learning for Online Nonlinear Control with Control Barrier Functions

Stable and Safe Reinforcement Learning via a Barrier-Lyapunov Actor-Critic Approach

Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints

Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm

Safe Q-learning for continuous-time linear systems

Model-Based Safe Reinforcement Learning With Time-Varying Constraints: Applications to Intelligent Vehicles

Safe Reinforcement Learning Using Robust Control Barrier Functions

Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning

Lyapunov-based uncertainty-aware safe reinforcement learning

Safe Online Integral Reinforcement Learning for Control Systems Via Controller Decomposition

Safe reinforcement learning for probabilistic reachability and safety specifications: A Lyapunov-based approach

Barrier Lyapunov Function-Based Safe Reinforcement Learning Algorithm for Autonomous Vehicles with System Uncertainty

Barrier Lyapunov Function-Based Safe Reinforcement Learning for Autonomous Vehicles with Optimized Backstepping

Safe Reinforcement Learning via a Model-Free Safety Certifier