Abstract:Mobile Manipulation (MM) systems are ideal candidates for taking up the role of a personal assistant in unstructured real-world environments. Among other challenges, MM requires effective coordination of the robot's embodiments for executing tasks that require both mobility and manipulation. Reinforcement Learning (RL) holds the promise of endowing robots with adaptive behaviors, but most methods require prohibitively large amounts of data for learning a useful control policy. In this work, we study the integration of robotic reachability priors in actor-critic RL methods for accelerating the learning of MM for reaching and fetching tasks. Namely, we consider the problem of optimal base placement and the subsequent decision of whether to activate the arm for reaching a 6D target. For this, we devise a novel Hybrid RL method that handles discrete and continuous actions jointly, resorting to the Gumbel-Softmax reparameterization. Next, we train a reachability prior using data from the operational robot workspace, inspired by classical methods. Subsequently, we derive Boosted Hybrid RL (BHyRL), a novel algorithm for learning Q-functions by modeling them as a sum of residual approximators. Every time a new task needs to be learned, we can transfer our learned residuals and learn the component of the Q-function that is task-specific, hence, maintaining the task structure from prior behaviors. Moreover, we find that regularizing the target policy with a prior policy yields more expressive behaviors. We evaluate our method in simulation in reaching and fetching tasks of increasing difficulty, and we show the superior performance of BHyRL against baseline methods. Finally, we zero-transfer our learned 6D fetching policy with BHyRL to our MM robot TIAGo++. For more details and code release, please refer to our project site: <a class="link-external link-http" href="http://irosalab.com/rlmmbp" rel="external noopener nofollow">this http URL</a>

On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer

A Comparison of Action Spaces for Learning Manipulation Tasks

Investigating the Impact of Choice on Deep Reinforcement Learning for Space Controls

Redundancy-aware Action Spaces for Robot Learning

Learning Task Space Actions for Bipedal Locomotion

A Comparative Study on State-Action Spaces for Learning Viewpoint Selection and Manipulation with Diffusion Policy

Sim-to-Real Transfer with Action Mapping and State Prediction for Robot Motion Control

Data-Driven Evaluation of Training Action Space for Reinforcement Learning

Variable Impedance Skill Learning for Contact-Rich Manipulation

Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approach

The Shortcomings of Force-from-Motion in Robot Learning

Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation

Active Exploration in Bayesian Model-based Reinforcement Learning for Robot Manipulation

Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning

Towards Real-World Efficiency: Domain Randomization in Reinforcement Learning for Pre-Capture of Free-Floating Moving Targets by Autonomous Robots

Robot Manipulation Skills Transfer for Sim-to-Real in Unstructured Environments

Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey

Sim-to-Real Transfer for Robotic Manipulation with Tactile Sensory

Robot Learning of Mobile Manipulation with Reachability Behavior Priors

Torque-Based Deep Reinforcement Learning for Task-and-Robot Agnostic Learning on Bipedal Robots Using Sim-to-Real Transfer

Guiding real-world reinforcement learning for in-contact manipulation tasks with Shared Control Templates