Visual Whole-Body Control for Legged Loco-Manipulation

Minghuan Liu,Zixuan Chen,Xuxin Cheng,Yandong Ji,Ri-Zhao Qiu,Ruihan Yang,Xiaolong Wang
2024-05-14
Abstract:We study the problem of mobile manipulation using legged robots equipped with an arm, namely legged loco-manipulation. The robot legs, while usually utilized for mobility, offer an opportunity to amplify the manipulation capabilities by conducting whole-body control. That is, the robot can control the legs and the arm at the same time to extend its workspace. We propose a framework that can conduct the whole-body control autonomously with visual observations. Our approach, namely Visual Whole-Body Control(VBC), is composed of a low-level policy using all degrees of freedom to track the body velocities along with the end-effector position, and a high-level policy proposing the velocities and end-effector position based on visual inputs. We train both levels of policies in simulation and perform Sim2Real transfer for real robot deployment. We perform extensive experiments and show significant improvements over baselines in picking up diverse objects in different configurations (heights, locations, orientations) and environments.
Robotics,Computer Vision and Pattern Recognition,Machine Learning
What problem does this paper attempt to address?
The paper aims to address the problem of mobile manipulation for legged robots (equipped with a manipulator), known as loco-manipulation. Specifically, the research goal is to enable the robot to autonomously perform whole-body control based on visual input, thereby picking up various objects at different heights and environments. The authors propose a method called "Visual Whole-Body Control" (VBC), which achieves this goal through a two-layer strategy: the low-level policy is responsible for tracking body velocity and end-effector position, while the high-level policy proposes these velocity and position commands based on visual input. This approach allows the robot to be trained in a simulated environment and directly applied to a real robot without additional fine-tuning, demonstrating significant improvements in different configurations (height, position, orientation) and environments. Moreover, the VBC method can handle complex terrains and exhibits retry behaviors in various tasks, enhancing operational flexibility and adaptability.