Abstract:Safe exploration aims at addressing the limitations of Reinforcement Learning (RL) in safety-critical scenarios, where failures during trial-and-error learning may incur high costs. Several methods exist to incorporate external knowledge or to use proximal sensor data to limit the exploration of unsafe states. However, reducing exploration risks in unknown environments, where an agent must discover safety threats during exploration, remains challenging. In this paper, we target the problem of safe exploration by guiding the training with counterexamples of the safety requirement. Our method abstracts both continuous and discrete state-space systems into compact abstract models representing the safety-relevant knowledge acquired by the agent during exploration. We then exploit probabilistic counterexample generation to construct minimal simulation submodels eliciting safety requirement violations, where the agent can efficiently train offline to refine its policy towards minimising the risk of safety violations during the subsequent online exploration. We demonstrate our method's effectiveness in reducing safety violations during online exploration in preliminary experiments by an average of 40.3% compared with QL and DQN standard algorithms and 29.1% compared with previous related work, while achieving comparable cumulative rewards with respect to unrestricted exploration and alternative approaches.

What problem does this paper attempt to address?

The paper primarily addresses the challenges faced when applying Reinforcement Learning (RL) in safety-critical scenarios, especially when the trial-and-error process during learning can lead to high costs or unsafe situations. To tackle this issue, the authors propose a method that uses counterexample-guided training to enhance the safety exploration capabilities of reinforcement learning. Specifically, the goals of the paper can be summarized as follows: 1. **Problem Background**: The core of reinforcement learning is a trial-and-error process where the agent learns the optimal policy through interaction with the environment. However, in safety-critical application scenarios, this trial-and-error process can lead to severe consequences, thus it is necessary to ensure that unsafe behaviors are avoided during the learning process. 2. **Solution Method**: - **Abstract Model Construction**: First, construct an abstract model to represent the safety-related knowledge acquired by the agent during exploration. This model can handle both continuous and discrete state space systems. - **Counterexample Generation**: Use probabilistic counterexample generation techniques to construct minimal simulation sub-models that reveal situations violating safety requirements. The agent can perform offline training in these sub-models to optimize its policy and reduce the risk of violating safety requirements during online exploration. - **Combining Offline and Online Learning**: By combining online exploration with counterexample-based offline training, gradually improve the agent's policy to explore unknown environments more safely. 3. **Experimental Results**: Preliminary experiments validated the effectiveness of this method. The results showed that this method, compared to standard Q-Learning and DQN algorithms as well as previous related work, improved the reduction of safety violations by an average of 40.3% and 29.1%, respectively, while maintaining cumulative rewards comparable to unrestricted exploration. In summary, this paper aims to enhance the safety of reinforcement learning in unknown or partially unknown environments through a novel counterexample-guided mechanism, effectively reducing potential risks, especially when facing safety-critical tasks.

Probabilistic Counterexample Guidance for Safer Reinforcement Learning (Extended Version)

Safe Sim-to-Real Robot Exploration with Constrained Bayesian Optimization

Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention

Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis

Almost Surely Safe Exploration and Exploitation for Deep Reinforcement Learning with State Safety Estimation

Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificate

State-Wise Safe Reinforcement Learning With Pixel Observations

ROSCOM: Robust Safe Reinforcement Learning on Stochastic Constraint Manifolds

Safe Exploration in Wireless Security: A Safe Reinforcement Learning Algorithm With Hierarchical Structure

Safe Reinforcement Learning by Imagining the Near Future

Provable Safe Reinforcement Learning with Binary Feedback

Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards

Counterexample-Guided Repair of Reinforcement Learning Systems Using Safety Critics

Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms

ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning

Safe reinforcement learning for probabilistic reachability and safety specifications: A Lyapunov-based approach

Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning

GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model

Evaluating Model-free Reinforcement Learning Toward Safety-critical Tasks

Do No Harm: A Counterfactual Approach to Safe Reinforcement Learning