Abstract:Autonomous off-road driving is challenging as risky actions taken by the robot may lead to catastrophic damage. As such, developing controllers in simulation is often desirable as it provides a safer and more economical alternative. However, accurately modeling robot dynamics is difficult due to the complex robot dynamics and terrain interactions in unstructured environments. Domain randomization addresses this problem by randomizing simulation dynamics parameters, however this approach sacrifices performance for robustness leading to policies that are sub-optimal for any target dynamics. We introduce a novel model-based reinforcement learning approach that aims to balance robustness with adaptability. Our approach trains a System Identification Transformer (SIT) and an Adaptive Dynamics Model (ADM) under a variety of simulated dynamics. The SIT uses attention mechanisms to distill state-transition observations from the target system into a context vector, which provides an abstraction for its target dynamics. Conditioned on this, the ADM probabilistically models the system's dynamics. Online, we use a Risk-Aware Model Predictive Path Integral controller (MPPI) to safely control the robot under its current understanding of the dynamics. We demonstrate in simulation as well as in multiple real-world environments that this approach enables safer behaviors upon initialization and becomes less conservative (i.e. faster) as its understanding of the target system dynamics improves with more observations. In particular, our approach results in an approximately 41% improvement in lap-time over the non-adaptive baseline while remaining safe across different environments.

Dynamics Adaptive Safe Reinforcement Learning with a Misspecified Simulator

Safe Sim-to-Real Robot Exploration with Constrained Bayesian Optimization

Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees

Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention

Safe Deep Policy Adaptation

Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints

Bypassing the Simulation-to-reality Gap: Online Reinforcement Learning using a Supervisor

Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificate

Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards

Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning

Safe and Sample-efficient Reinforcement Learning for Clustered Dynamic Environments

Offline Imitation Learning with a Misspecified Simulator.

Context-Aware Safe Reinforcement Learning for Non-Stationary Environments

Progressive Adaptive Chance-Constrained Safeguards for Reinforcement Learning.

GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model

Dual Action Policy for Robust Sim-to-Real Reinforcement Learning

Pay Attention to How You Drive: Safe and Adaptive Model-Based Reinforcement Learning for Off-Road Driving

Sim-to-Real Transfer with Action Mapping and State Prediction for Robot Motion Control

Sim-to-Real Interactive Recommendation via Off-Dynamics Reinforcement Learning

Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance

Safe Reinforcement Learning for Dynamical Systems Using Barrier Certificates