Abstract:We envision a telepresence system that enhances remote work by facilitating both physical and immersive visual interactions between individuals. However, during robot teleoperation, communication often lacks realism, as users see the robot's body rather than the remote individual. To address this, we propose a method for overlaying a digital human model onto a humanoid robot using XR visualization, enabling an immersive 3D telepresence experience. Our approach employs a learning-based method to estimate the 2D poses of the humanoid robot from head-worn stereo views, leveraging a newly collected dataset of full-body poses for humanoid robots. The stereo 2D poses and sparse inertial measurements from the remote operator are optimized to compute 3D poses over time. The digital human is localized from the perspective of a continuously moving observer, utilizing the estimated 3D pose of the humanoid robot. Our moving camera-based pose estimation method does not rely on any markers or external knowledge of the robot's status, effectively overcoming challenges such as marker occlusion, calibration issues, and dependencies on headset tracking errors. We demonstrate the system in a remote physical training scenario, achieving real-time performance at 40 fps, which enables simultaneous immersive and physical interactions. Experimental results show that our learning-based 3D pose estimation method, which operates without prior knowledge of the robot, significantly outperforms alternative approaches requiring the robot's global pose, particularly during rapid headset movements, achieving markerless digital human augmentation from head-worn views.

Design of a human-robot interaction system for robot teleoperation based on digital twinning

Revitalizing Human-Robot Interaction: Phygital Twin Driven Robot Avatar for China–Sweden Teleoperation

The Design and Implementation of Natural Human-Robot Interaction System Based on Kinect Sensor

Design of Kinect-Based Human Robot Interaction Systems for A Robocup Middle Size League Soccer Robot

Development of a Synchronized Human-Robot-Virtuality Interaction System Using Cooperative Robot and Motion Capture Device

Interface Design of a Human-Robot Interaction System for Dual-Manipulators Teleoperation Based on Virtual Reality *

A Universal Self-Adaption Workspace Mapping Method for Human–Robot Interaction Using Kinect Sensor Data

Enabling collaborative assembly between humans and robots using a digital twin system

Human-robot Interaction Oriented Human-in-the-loop Real-time Motion Imitation on a Humanoid Tri-Co Robot

An Intelligent Interaction Framework for Teleoperation Based on Human-Machine Cooperation

A framework and method for Human-Robot cooperative safe control based on digital twin

An Integrated Vision-Based System for Efficient Robot Arm Teleoperation

Intelligent robotic arm for human pose recognition based on teleoperation system

Telemanipulation Via Internet Based on Human-Robot Cooperation

Motion Similarity Evaluation between Human and a Tri-Co Robot during Real-Time Imitation with a Trajectory Dynamic Time Warping Model

A Vision-based Human Digital Twin Modelling Approach for Adaptive Human-Robot Collaboration

Design and implementation of a dexterous human-robot interaction system based on haptic shared control

Full-Body Pose Estimation of Humanoid Robots Using Head-Worn Cameras for Digital Human-Augmented Robotic Telepresence

Digital Twin-Driven Human Robot Collaboration Using a Digital Human

An intuitive human robot interface for tele-operation

Whole-Body Teleoperation Control of Dual-Arm Robot Using Sensor Fusion