3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features

Chenfeng Xu,Huan Ling,Sanja Fidler,Or Litany

2023-11-08

Abstract:We present 3DiffTection, a state-of-the-art method for 3D object detection from single images, leveraging features from a 3D-aware diffusion model. Annotating large-scale image data for 3D detection is resource-intensive and time-consuming. Recently, pretrained large image diffusion models have become prominent as effective feature extractors for 2D perception tasks. However, these features are initially trained on paired text and image data, which are not optimized for 3D tasks, and often exhibit a domain gap when applied to the target data. Our approach bridges these gaps through two specialized tuning strategies: geometric and semantic. For geometric tuning, we fine-tune a diffusion model to perform novel view synthesis conditioned on a single image, by introducing a novel epipolar warp operator. This task meets two essential criteria: the necessity for 3D awareness and reliance solely on posed image data, which are readily available (e.g., from videos) and does not require manual annotation. For semantic refinement, we further train the model on target data with detection supervision. Both tuning phases employ ControlNet to preserve the integrity of the original feature capabilities. In the final step, we harness these enhanced capabilities to conduct a test-time prediction ensemble across multiple virtual viewpoints. Through our methodology, we obtain 3D-aware features that are tailored for 3D detection and excel in identifying cross-view point correspondences. Consequently, our model emerges as a powerful 3D detector, substantially surpassing previous benchmarks, e.g., Cube-RCNN, a precedent in single-view 3D detection by 9.43\% in AP3D on the Omni3D-ARkitscene dataset. Furthermore, 3DiffTection showcases robust data efficiency and generalization to cross-domain data.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the paper attempts to solve

The paper "3DIFFTECTION: 3D Object Detection Based on Geometric - Aware Diffusion Features" aims to solve the problem of 3D object detection from a single image. Specifically, the paper attempts to solve the following key problems:

1. **Cost and efficiency of labeled data**:
- 3D object detection from a single image requires a large amount of labeled data, which is both time - consuming and expensive. The paper proposes a method of using pre - trained large - scale image diffusion models to extract features. These models were initially trained on paired text and image data, but are not suitable for 3D tasks, so there is a domain gap.

2. **Transfer of 2D features to 3D tasks**:
- Although pre - trained diffusion models perform well in 2D perception tasks, they do not work well when directly applied to 3D tasks because these models lack 3D perception ability. The paper bridges this gap through two specialized tuning strategies (geometric tuning and semantic tuning).

3. **Improving 3D detection performance**:
- Through geometric tuning, the paper introduces a new epipolar warp operator, enabling the diffusion model to generate new views given a single image. Through semantic tuning, the model is further trained on the target data to optimize the detection performance. Finally, the accuracy of 3D detection is improved by performing test - time prediction integration on multiple virtual views.

4. **Generalization ability of cross - domain data**:
- The paper also demonstrates the generalization ability of the model on different datasets, proving its robustness and efficiency on cross - domain data.

### Main contributions

1. **Enhancing the 3D perception ability of pre - trained 2D diffusion models**:
- Through the view synthesis task, the paper proposes a scalable technique to make pre - trained 2D diffusion models have 3D perception ability.

2. **Adapting to 3D detection tasks and target domains**:
- Using 3D - enhanced features, the paper trains a standard detection head and further adapts to the target task and dataset through semantic ControlNet.

3. **Improving detection performance by using view synthesis ability**:
- By performing prediction integration on multiple synthesized views, the performance of 3D detection is further improved.

### Experimental results

The paper conducted experiments on the Omni3D - ARKitScenes dataset. The results show that 3DiffTection significantly outperforms previous benchmark methods, such as Cube - RCNN, in terms of the AP3D metric, with an improvement of 7.4% at a resolution of 256 × 256 and 9.43% at a resolution of 512 × 512. In addition, the paper also demonstrates the good generalization ability of the model on cross - domain datasets.

### Summary

Through innovative geometric and semantic tuning strategies, this paper successfully applies pre - trained 2D diffusion models to 3D object detection tasks, significantly improves the detection performance, and shows good data efficiency and cross - domain generalization ability.