Abstract:Deep reinforcement learning (DRL) has become a dominant deep-learning paradigm for tasks where complex policies are learned within reactive systems. Unfortunately, these policies are known to be susceptible to bugs. Despite significant progress in DNN verification, there has been little work demonstrating the use of modern verification tools on real-world, DRL-controlled systems. In this case study, we attempt to begin bridging this gap, and focus on the important task of mapless robotic navigation -- a classic robotics problem, in which a robot, usually controlled by a DRL agent, needs to efficiently and safely navigate through an unknown arena towards a target. We demonstrate how modern verification engines can be used for effective model selection, i.e., selecting the best available policy for the robot in question from a pool of candidate policies. Specifically, we use verification to detect and rule out policies that may demonstrate suboptimal behavior, such as collisions and infinite loops. We also apply verification to identify models with overly conservative behavior, thus allowing users to choose superior policies, which might be better at finding shorter paths to a target. To validate our work, we conducted extensive experiments on an actual robot, and confirmed that the suboptimal policies detected by our method were indeed flawed. We also demonstrate the superiority of our verification-driven approach over state-of-the-art, gradient attacks. Our work is the first to establish the usefulness of DNN verification in identifying and filtering out suboptimal DRL policies in real-world robots, and we believe that the methods presented here are applicable to a wide range of systems that incorporate deep-learning-based agents.

Efficient LTL Model Checking of Deep Reinforcement Learning Systems Using Policy Extraction

Automatically ‘verifying’ Discrete-Time Complex Systems Through Learning, Abstraction and Refinement

Learning on Abstract Domains: A New Approach for Verifiable Guarantee in Reinforcement Learning

TRAINIFY: A CEGAR-Driven Training and Verification Framework for Safe Deep Reinforcement Learning

Boosting Verification of Deep Reinforcement Learning Via Piece-Wise Linear Decision Neural Networks.

Robustness Verification of Deep Reinforcement Learning Based Control Systems using Reward Martingales

Safe Reinforcement Learning for CPSs via Formal Modeling and Verification

Deep Reinforcement Learning with Temporal Logics

Formally Verifying Deep Reinforcement Learning Controllers with Lyapunov Barrier Certificates

Dependable Reinforcement Learning Via Timed Differential Dynamic Logic.

Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous Systems

Probabilistic Model Checking of Stochastic Reinforcement Learning Policies

Certified Reinforcement Learning with Logic Guidance

Verifying ReLU Neural Networks from a Model Checking Perspective

LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learning

Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guarantees

Verifiable Reinforcement Learning via Policy Extraction

Mission-driven Exploration for Accelerated Deep Reinforcement Learning with Temporal Logic Task Specifications

Verifiable RNN-Based Policies for POMDPs Under Temporal Logic Constraints

Verifying Learning-Based Robotic Navigation Systems

Evaluating the Safety of Deep Reinforcement Learning Models using Semi-Formal Verification