Abstract:Deep reinforcement learning (DRL) has become a dominant deep-learning paradigm for tasks where complex policies are learned within reactive systems. Unfortunately, these policies are known to be susceptible to bugs. Despite significant progress in DNN verification, there has been little work demonstrating the use of modern verification tools on real-world, DRL-controlled systems. In this case study, we attempt to begin bridging this gap, and focus on the important task of mapless robotic navigation -- a classic robotics problem, in which a robot, usually controlled by a DRL agent, needs to efficiently and safely navigate through an unknown arena towards a target. We demonstrate how modern verification engines can be used for effective model selection, i.e., selecting the best available policy for the robot in question from a pool of candidate policies. Specifically, we use verification to detect and rule out policies that may demonstrate suboptimal behavior, such as collisions and infinite loops. We also apply verification to identify models with overly conservative behavior, thus allowing users to choose superior policies, which might be better at finding shorter paths to a target. To validate our work, we conducted extensive experiments on an actual robot, and confirmed that the suboptimal policies detected by our method were indeed flawed. We also demonstrate the superiority of our verification-driven approach over state-of-the-art, gradient attacks. Our work is the first to establish the usefulness of DNN verification in identifying and filtering out suboptimal DRL policies in real-world robots, and we believe that the methods presented here are applicable to a wide range of systems that incorporate deep-learning-based agents.

TRAINIFY: A CEGAR-Driven Training and Verification Framework for Safe Deep Reinforcement Learning

Reinfier and Reintrainer: Verification and Interpretation-Driven Safe Deep Reinforcement Learning Frameworks

Automatically ‘verifying’ Discrete-Time Complex Systems Through Learning, Abstraction and Refinement

Safe and Reliable Training of Learning-Based Aerospace Controllers

Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous Systems

Formally Verifying Deep Reinforcement Learning Controllers with Lyapunov Barrier Certificates

Robustness Verification of Deep Reinforcement Learning Based Control Systems using Reward Martingales

Boosting Verification of Deep Reinforcement Learning Via Piece-Wise Linear Decision Neural Networks.

Verification-Guided Shielding for Deep Reinforcement Learning

Efficient LTL Model Checking of Deep Reinforcement Learning Systems Using Policy Extraction

Scalable Synthesis of Verified Controllers in Deep Reinforcement Learning

Verified Safe Reinforcement Learning for Neural Network Dynamic Models

Train Trajectory Optimization with High-Risk State Space Boundaries: A Safe Reinforcement Learning Approach

Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning

Verifying Learning-Based Robotic Navigation Systems

Value Functions are Control Barrier Functions: Verification of Safe Policies using Control Theory

Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillation

Training Verifiably Robust Agents Using Set-Based Reinforcement Learning

Safe Reinforcement Learning via Formal Methods: Toward Safe Control Through Proof and Learning

Dense reinforcement learning for safety validation of autonomous vehicles

Safe Reinforcement Learning Using Robust Control Barrier Functions