Abstract:A critical bottleneck limiting imitation learning in robotics is the lack of data. This problem is more severe in mobile manipulation, where collecting demonstrations is harder than in stationary manipulation due to the lack of available and easy-to-use teleoperation interfaces. In this work, we demonstrate TeleMoMa, a general and modular interface for whole-body teleoperation of mobile manipulators. TeleMoMa unifies multiple human interfaces including RGB and depth cameras, virtual reality controllers, keyboard, joysticks, etc., and any combination thereof. In its more accessible version, TeleMoMa works using simply vision (e.g., an RGB-D camera), lowering the entry bar for humans to provide mobile manipulation demonstrations. We demonstrate the versatility of TeleMoMa by teleoperating several existing mobile manipulators - PAL Tiago++, Toyota HSR, and Fetch - in simulation and the real world. We demonstrate the quality of the demonstrations collected with TeleMoMa by training imitation learning policies for mobile manipulation tasks involving synchronized whole-body motion. Finally, we also show that TeleMoMa's teleoperation channel enables teleoperation on site, looking at the robot, or remote, sending commands and observations through a computer network, and perform user studies to evaluate how easy it is for novice users to learn to collect demonstrations with different combinations of human interfaces enabled by our system. We hope TeleMoMa becomes a helpful tool for the community enabling researchers to collect whole-body mobile manipulation demonstrations. For more information and video results, https://robin-lab.cs.utexas.edu/telemoma-web.

Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation

Learning Visual Quadrupedal Loco-Manipulation from Demonstrations

Opt2Skill: Imitating Dynamically-feasible Whole-Body Trajectories for Versatile Humanoid Loco-Manipulation

Wheeled Humanoid Bilateral Teleoperation with Position-Force Control Modes for Dynamic Loco-Manipulation

Human-in-the-Loop Imitation Learning using Remote Teleoperation

Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation

Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning

TeLeMan: Teleoperation for Legged Robot Loco-Manipulation using Wearable IMU-based Motion Capture

Real-Time Whole-Body Imitation by Humanoid Robots and Task-Oriented Teleoperation Using an Analytical Mapping Method and Quantitative Evaluation

Human-robot Interaction Oriented Human-in-the-loop Real-time Motion Imitation on a Humanoid Tri-Co Robot

A Combined Learning and Optimization Framework to Transfer Human Whole-body Loco-manipulation Skills to Mobile Manipulators

HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulation

Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning

Behavior Imitation for Manipulator Control and Grasping with Deep Reinforcement Learning

Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition

Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration

Grasping Diverse Objects with Simulated Humanoids

Haptic-ACT: Bridging Human Intuition with Compliant Robotic Manipulation via Immersive VR

TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation

Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking with a Wheeled-Quadrupedal-Manipulator

MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation