3D Diffuser Actor: Policy Diffusion with 3D Scene Representations

Tsung-Wei Ke,Nikolaos Gkanatsios,Katerina Fragkiadaki

2024-07-25

Abstract:Diffusion policies are conditional diffusion models that learn robot action distributions conditioned on the robot and environment state. They have recently shown to outperform both deterministic and alternative action distribution learning formulations. 3D robot policies use 3D scene feature representations aggregated from a single or multiple camera views using sensed depth. They have shown to generalize better than their 2D counterparts across camera viewpoints. We unify these two lines of work and present 3D Diffuser Actor, a neural policy equipped with a novel 3D denoising transformer that fuses information from the 3D visual scene, a language instruction and proprioception to predict the noise in noised 3D robot pose trajectories. 3D Diffuser Actor sets a new state-of-the-art on RLBench with an absolute performance gain of 18.1% over the current SOTA on a multi-view setup and an absolute gain of 13.1% on a single-view setup. On the CALVIN benchmark, it improves over the current SOTA by a 9% relative increase. It also learns to control a robot manipulator in the real world from a handful of demonstrations. Through thorough comparisons with the current SOTA policies and ablations of our model, we show 3D Diffuser Actor's design choices dramatically outperform 2D representations, regression and classification objectives, absolute attentions, and holistic non-tokenized 3D scene embeddings.

Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the problem of combining 3D scene representation with diffusion models in robotic manipulation tasks to enhance the performance of multimodal action prediction. Specifically, the authors propose a new method called 3D Diffuser Actor, which is a neural policy model that predicts noise in robot pose trajectories by fusing information from 3D visual scenes, language instructions, and proprioception through a novel 3D denoising transformer. This method achieves significantly better results than existing methods in RLBench and CALVIN benchmarks and demonstrates its multi-task manipulation capabilities in the real world.

The main contribution of the 3D Diffuser Actor lies in combining 3D scene representation with diffusion objectives for learning robotic policies from examples. This approach not only surpasses traditional 2D representations and other action distribution learning methods but also experimentally confirms the clear advantages of its design choices in handling multimodal action prediction.