Abstract:Reinforcement learning (RL) has revolutionized decision-making across a wide range of domains over the past few decades. Yet, deploying RL policies in real-world scenarios presents the crucial challenge of ensuring safety. Traditional safe RL approaches have predominantly focused on incorporating predefined safety constraints into the policy learning process. However, this reliance on predefined safety constraints poses limitations in dynamic and unpredictable real-world settings where such constraints may not be available or sufficiently adaptable. Bridging this gap, we propose a novel approach that concurrently learns a safe RL control policy and identifies the unknown safety constraint parameters of a given environment. Initializing with a parametric signal temporal logic (pSTL) safety specification and a small initial labeled dataset, we frame the problem as a bilevel optimization task, intricately integrating constrained policy optimization, using a Lagrangian-variant of the twin delayed deep deterministic policy gradient (TD3) algorithm, with Bayesian optimization for optimizing parameters for the given pSTL safety specification. Through experimentation in comprehensive case studies, we validate the efficacy of this approach across varying forms of environmental constraints, consistently yielding safe RL policies with high returns. Furthermore, our findings indicate successful learning of STL safety constraint parameters, exhibiting a high degree of conformity with true environmental safety constraints. The performance of our model closely mirrors that of an ideal scenario that possesses complete prior knowledge of safety constraints, demonstrating its proficiency in accurately identifying environmental safety constraints and learning safe policies that adhere to those constraints.

State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards

Convergence Rate of Primal-Dual Approach to Constrained Reinforcement Learning with Softmax Policy

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

Multiagent Reinforcement Learning for Strictly Constrained Tasks Based on Reward Recorder

A Stochastic Composite Augmented Lagrangian Method for Reinforcement Learning.

Multi-agent assignment via state augmented reinforcement learning

Resilient Constrained Reinforcement Learning

A Dual Perspective of Reinforcement Learning for Imposing Policy Constraints

Evolving Constrained Reinforcement Learning Policy

Near-Optimal Solutions of Constrained Learning Problems

Direct Behavior Specification via Constrained Reinforcement Learning

Probabilistic Constraint for Safety-Critical Reinforcement Learning

Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning

Confronting Reward Model Overoptimization with Constrained RLHF

Assured RL: Reinforcement Learning with Almost Sure Constraints

On Bellman's principle of optimality and Reinforcement learning for safety-constrained Markov decision process

Truly No-Regret Learning in Constrained MDPs

Constrained Reinforcement Learning with Smoothed Log Barrier Function

Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning

Constrained Reinforcement Learning with Average Reward Objective: Model-Based and Model-Free Algorithms