Abstract:The problem of uncertainty is a feature of real world robotics problems and any control framework must contend with it in order to succeed in real applications tasks. Reinforcement Learning is no different, and epistemic uncertainty arising from model uncertainty or misspecification is a challenge well captured by the sim-to-real gap. A simple solution to this issue is domain randomization (DR), which unfortunately can result in conservative agents. As a remedy to this conservativeness, the use of universal policies that take additional information about the randomized domain has risen as an alternative solution, along with recurrent neural network-based controllers. Uncertainty-aware universal policies present a particularly compelling solution able to account for system identification uncertainties during deployment. In this paper, we reveal that the challenge of efficiently optimizing uncertainty-aware policies can be fundamentally reframed as solving the convex coverage set (CCS) problem within a multi-objective reinforcement learning (MORL) context. By introducing a novel Markov decision process (MDP) framework where each domain's performance is treated as an independent objective, we unify the training of uncertainty-aware policies with MORL approaches. This connection enables the application of MORL algorithms for domain randomization (DR), allowing for more efficient policy optimization. To illustrate this, we focus on the linear utility function, which aligns with the expectation in DR formulations, and propose a series of algorithms adapted from the MORL literature to solve the CCS, demonstrating their ability to enhance the performance of uncertainty-aware policies.

Domain Randomization for Robust, Affordable and Effective Closed-loop Control of Soft Robots

Continual Domain Randomization

Reinforcement Learning Controllers for Soft Robots using Learned Environments

DR2L: Surfacing Corner Cases to Robustify Autonomous Driving via Domain Randomization Reinforcement Learning

Adaptability Preserving Domain Decomposition for Stabilizing Sim2Real Reinforcement Learning

Towards Real-World Efficiency: Domain Randomization in Reinforcement Learning for Pre-Capture of Free-Floating Moving Targets by Autonomous Robots

Open-Loop Motion Control of a Hydraulic Soft Robotic Arm Using Deep Reinforcement Learning

Domain Randomization via Entropy Maximization

Domains as Objectives: Domain-Uncertainty-Aware Policy Optimization through Explicit Multi-Domain Convex Coverage Set Learning

RL-based Adaptive Controller for High Precision Reaching in a Soft Robot Arm

Transferable Adversarial Attacks on Deep Reinforcement Learning with Domain Randomization

Task and Domain Adaptive Reinforcement Learning for Robot Control

Reinforcement Learning-Based Bionic Reflex Control for Anthropomorphic Robotic Grasping exploiting Domain Randomization

Domain Randomization for Sim2real Transfer of Automatically Generated Grasping Datasets

Leveraging Randomized Smoothing for Optimal Control of Nonsmooth Dynamical Systems

DROID: Minimizing the Reality Gap Using Single-Shot Human Demonstration

Understanding Domain Randomization for Sim-to-real Transfer

Robust Learning-Based Control for Uncertain Nonlinear Systems With Validation on a Soft Robot

Deep Reinforcement Learning with Domain Randomization for Overhead Crane Control with Payload Mass Variations

Elastica: A Compliant Mechanics Environment for Soft Robotic Control