Abstract:Reinforcement learning (RL) has revolutionized decision-making across a wide range of domains over the past few decades. Yet, deploying RL policies in real-world scenarios presents the crucial challenge of ensuring safety. Traditional safe RL approaches have predominantly focused on incorporating predefined safety constraints into the policy learning process. However, this reliance on predefined safety constraints poses limitations in dynamic and unpredictable real-world settings where such constraints may not be available or sufficiently adaptable. Bridging this gap, we propose a novel approach that concurrently learns a safe RL control policy and identifies the unknown safety constraint parameters of a given environment. Initializing with a parametric signal temporal logic (pSTL) safety specification and a small initial labeled dataset, we frame the problem as a bilevel optimization task, intricately integrating constrained policy optimization, using a Lagrangian-variant of the twin delayed deep deterministic policy gradient (TD3) algorithm, with Bayesian optimization for optimizing parameters for the given pSTL safety specification. Through experimentation in comprehensive case studies, we validate the efficacy of this approach across varying forms of environmental constraints, consistently yielding safe RL policies with high returns. Furthermore, our findings indicate successful learning of STL safety constraint parameters, exhibiting a high degree of conformity with true environmental safety constraints. The performance of our model closely mirrors that of an ideal scenario that possesses complete prior knowledge of safety constraints, demonstrating its proficiency in accurately identifying environmental safety constraints and learning safe policies that adhere to those constraints.

Mixed Reinforcement Learning for Efficient Policy Optimization in Stochastic Environments

Mixed Reinforcement Learning with Additive Stochastic Uncertainty

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Model-Based Robot Learning Control with Uncertainty Directed Exploration

Safe Reinforcement Learning-Based Eco-Driving Control for Mixed Traffic Flows With Disturbances

Non-local Policy Optimization via Diversity-regularized Collaborative Exploration

OMPO: A Unified Framework for RL under Policy and Dynamics Shifts

Model-Based Safe Reinforcement Learning With Time-Varying Constraints: Applications to Intelligent Vehicles

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning

Blending Imitation and Reinforcement Learning for Robust Policy Improvement

Control in Stochastic Environment with Delays: A Model-based Reinforcement Learning Approach

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

State Regularized Policy Optimization on Data with Dynamics Shift

Optimal Tracking Control of Nonlinear Multiagent Systems Using Internal Reinforce Q-Learning

Multiple Suboptimal Policies Integrated Reinforcement Learning Algorithm for Path Planning

A Stochastic Trust-Region Framework for Policy Optimization

Mixed Policy Gradient: off-policy reinforcement learning driven jointly by data and model

A Recurrent Reinforcement Learning Approach Applicable To Highly Uncertain Environments

Mixed-Integer Optimal Control via Reinforcement Learning: A Case Study on Hybrid Electric Vehicle Energy Management

Model-Based Reinforcement Learning via Stochastic Hybrid Models