Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning
Haoyi Zhu, Yating Wang, Di Huang, Weicai Ye, Wanli Ouyang, Tong He
2024-02-05
Abstract:In this study, we explore the influence of different observation spaces on
robot learning, focusing on three predominant modalities: RGB, RGB-D, and point
cloud. Through extensive experimentation on over 17 varied contact-rich
manipulation tasks, conducted across two benchmarks and simulators, we have
observed a notable trend: point cloud-based methods, even those with the
simplest designs, frequently surpass their RGB and RGB-D counterparts in
performance. This remains consistent in both scenarios: training from scratch
and utilizing pretraining. Furthermore, our findings indicate that point cloud
observations lead to improved policy zero-shot generalization in relation to
various geometry and visual clues, including camera viewpoints, lighting
conditions, noise levels and background appearance. The outcomes suggest that
3D point cloud is a valuable observation modality for intricate robotic tasks.
We will open-source all our codes and checkpoints, hoping that our insights can
help design more generalizable and robust robotic models.
Robotics,Machine Learning,Computer Vision and Pattern Recognition,Artificial Intelligence