Abstract:Robotic manipulation requires accurate perception of the environment, which poses a significant challenge due to its inherent complexity and constantly changing nature. In this context, RGB image and point-cloud observations are two commonly used modalities in visual-based robotic manipulation, but each of these modalities have their own limitations. Commercial point-cloud observations often suffer from issues like sparse sampling and noisy output due to the limits of the emission-reception imaging principle. On the other hand, RGB images, while rich in texture information, lack essential depth and 3D information crucial for robotic manipulation. To mitigate these challenges, we propose an image-only robotic manipulation framework that leverages an eye-on-hand monocular camera installed on the robot's parallel gripper. By moving with the robot gripper, this camera gains the ability to actively perceive object from multiple perspectives during the manipulation process. This enables the estimation of 6D object poses, which can be utilized for manipulation. While, obtaining images from more and diverse viewpoints typically improves pose estimation, it also increases the manipulation time. To address this trade-off, we employ a reinforcement learning policy to synchronize the manipulation strategy with active perception, achieving a balance between 6D pose accuracy and manipulation efficiency. Our experimental results in both simulated and real-world environments showcase the state-of-the-art effectiveness of our approach. %, which, to the best of our knowledge, is the first to achieve robust real-world robotic manipulation through active pose estimation. We believe that our method will inspire further research on real-world-oriented robotic manipulation.

Leveraging RGB-Pressure for Whole-body Human-to-Humanoid Motion Imitation

Motion Imitation of a Humanoid Robot Via Pose Estimation

Beyond End-Effector: Utilizing High-Resolution Tactile Signals for Physical Human–Robot Interaction

Visual Perception Method Based on Human Pose Estimation for Humanoid Robot Imitating Human Motions.

Human Motion Capture Using Wireless Inertial Sensors

Leveraging depth cameras and wearable pressure sensors for full-body kinematics and dynamics capture

An Attention-Based Deep Learning Approach for Inertial Motion Recognition and Estimation in Human-Robot Collaboration

Real-Time Whole-Body Imitation by Humanoid Robots and Task-Oriented Teleoperation Using an Analytical Mapping Method and Quantitative Evaluation

Human Motion Tracking by Multiple RGBD Cameras.

Human Motion Tracking with Less Constraint of Initial Posture from a Single RGB-D Sensor

Whole-body humanoid robot imitation with pose similarity evaluation

Expressive Whole-Body Control for Humanoid Robots

PressureVision: Estimating Hand Pressure from a Single RGB Image

RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation

Learning Humanoid Locomotion with Perceptive Internal Model

Motion Capture Research: 3D Human Pose Recovery Based on RGB Video Sequences

I-CTRL: Imitation to Control Humanoid Robots Through Constrained Reinforcement Learning

Robust Model-Based 3D Torso Pose Estimation in RGB-D Sequences

Resolving Position Ambiguity of IMU-Based Human Pose with a Single RGB Camera

Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation

Ubiquitous Human Body Motion Capture Using Micro-Sensors