Abstract:Reinforcement learning (RL) has revolutionized decision-making across a wide range of domains over the past few decades. Yet, deploying RL policies in real-world scenarios presents the crucial challenge of ensuring safety. Traditional safe RL approaches have predominantly focused on incorporating predefined safety constraints into the policy learning process. However, this reliance on predefined safety constraints poses limitations in dynamic and unpredictable real-world settings where such constraints may not be available or sufficiently adaptable. Bridging this gap, we propose a novel approach that concurrently learns a safe RL control policy and identifies the unknown safety constraint parameters of a given environment. Initializing with a parametric signal temporal logic (pSTL) safety specification and a small initial labeled dataset, we frame the problem as a bilevel optimization task, intricately integrating constrained policy optimization, using a Lagrangian-variant of the twin delayed deep deterministic policy gradient (TD3) algorithm, with Bayesian optimization for optimizing parameters for the given pSTL safety specification. Through experimentation in comprehensive case studies, we validate the efficacy of this approach across varying forms of environmental constraints, consistently yielding safe RL policies with high returns. Furthermore, our findings indicate successful learning of STL safety constraint parameters, exhibiting a high degree of conformity with true environmental safety constraints. The performance of our model closely mirrors that of an ideal scenario that possesses complete prior knowledge of safety constraints, demonstrating its proficiency in accurately identifying environmental safety constraints and learning safe policies that adhere to those constraints.

Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret

Safe Reinforcement Learning Using Finite-Horizon Gradient-based Estimation

Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning

Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayesian Theory

Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting

Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning

Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics Via Dual-agent Reinforcement Learning

Safe reinforcement learning for probabilistic reachability and safety specifications: A Lyapunov-based approach

Joint Learning of Policy with Unknown Temporal Constraints for Safe Reinforcement Learning

Safe Reinforcement Learning by Imagining the Near Future

Scaling life-long off-policy learning

Model-Based Safe Reinforcement Learning With Time-Varying Constraints: Applications to Intelligent Vehicles

Stronger Regret Bounds for Safe Online Reinforcement Learning in the Linear Quadratic Regulator

Safe Reinforcement Learning for Dynamical Systems Using Barrier Certificates

End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks

Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning

Langevin Policy for Safe Reinforcement Learning

Lyapunov-based uncertainty-aware safe reinforcement learning

Sublinear Regret for a Class of Continuous-Time Linear--Quadratic Reinforcement Learning Problems

Safe Reinforcement Learning with Dead-Ends Avoidance and Recovery

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles