View-Invariant Policy Learning via Zero-Shot Novel View Synthesis

Stephen Tian,Blake Wulfe,Kyle Sargent,Katherine Liu,Sergey Zakharov,Vitor Guizilini,Jiajun Wu

2024-09-06

Abstract:Large-scale visuomotor policy learning is a promising approach toward developing generalizable manipulation systems. Yet, policies that can be deployed on diverse embodiments, environments, and observational modalities remain elusive. In this work, we investigate how knowledge from large-scale visual data of the world may be used to address one axis of variation for generalizable manipulation: observational viewpoint. Specifically, we study single-image novel view synthesis models, which learn 3D-aware scene-level priors by rendering images of the same scene from alternate camera viewpoints given a single input image. For practical application to diverse robotic data, these models must operate zero-shot, performing view synthesis on unseen tasks and environments. We empirically analyze view synthesis models within a simple data-augmentation scheme that we call View Synthesis Augmentation (VISTA) to understand their capabilities for learning viewpoint-invariant policies from single-viewpoint demonstration data. Upon evaluating the robustness of policies trained with our method to out-of-distribution camera viewpoints, we find that they outperform baselines in both simulated and real-world manipulation tasks. Videos and additional visualizations are available at <a class="link-external link-https" href="https://s-tian.github.io/projects/vista" rel="external noopener nofollow">this https URL</a>.

Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The paper aims to address the generalization problem of robotic vision strategies under different observation perspectives. Specifically, researchers have found that strategies trained under fixed perspectives struggle to work effectively under other perspectives. To overcome this challenge, they propose a method called "View Synthesis Augmentation (VISTA)." This method enhances the input data during training using a novel view synthesis model from a single image, enabling the strategy to maintain robustness under different camera perspectives.

The paper validates the effectiveness of the proposed method through a series of experiments, including both simulated environments and real-world robotic tasks. The results show that strategies trained with VISTA outperform baseline methods when handling unseen perspectives. Additionally, after fine-tuning the novel view synthesis model on a large-scale real robot dataset, the performance is further improved. The study also explores the effect of combining wrist camera observations to reduce the observation domain gap, finding that this combination can complementarily enhance performance. In summary, VISTA provides a simple and effective way to enhance the robustness and generalization ability of robotic strategies under different perspectives.