Abstract:We present 3DiffTection, a state-of-the-art method for 3D object detection from single images, leveraging features from a 3D-aware diffusion model. Annotating large-scale image data for 3D detection is resource-intensive and time-consuming. Recently, pretrained large image diffusion models have become prominent as effective feature extractors for 2D perception tasks. However, these features are initially trained on paired text and image data, which are not optimized for 3D tasks, and often exhibit a domain gap when applied to the target data. Our approach bridges these gaps through two specialized tuning strategies: geometric and semantic. For geometric tuning, we fine-tune a diffusion model to perform novel view synthesis conditioned on a single image, by introducing a novel epipolar warp operator. This task meets two essential criteria: the necessity for 3D awareness and reliance solely on posed image data, which are readily available (e.g., from videos) and does not require manual annotation. For semantic refinement, we further train the model on target data with detection supervision. Both tuning phases employ ControlNet to preserve the integrity of the original feature capabilities. In the final step, we harness these enhanced capabilities to conduct a test-time prediction ensemble across multiple virtual viewpoints. Through our methodology, we obtain 3D-aware features that are tailored for 3D detection and excel in identifying cross-view point correspondences. Consequently, our model emerges as a powerful 3D detector, substantially surpassing previous benchmarks, e.g., Cube-RCNN, a precedent in single-view 3D detection by 9.43\% in AP3D on the Omni3D-ARkitscene dataset. Furthermore, 3DiffTection showcases robust data efficiency and generalization to cross-domain data.

A Divergence Mean-Based Geometric Detector with a Pre-Processing Procedure

A Divergence Median-based Geometric Detector with A Weighted Averaging Filter

Target Detection Within Nonhomogeneous Clutter Via Total Bregman Divergence-Based Matrix Information Geometry Detectors

Geometric Target Detection Based on Total Bregman Divergence

Infrared small target detection based on divergence operator and nonlinear classifier

Information Geometry for Covariance Estimation in Heterogeneous Clutter with Total Bregman Divergence

Matrix Cfar Detectors Based on Symmetrized Kullback-Leibler and Total Kullback-Leibler Divergences

Persymmetric Subspace Gradient-based Detectors for Distributed Targets in Homogeneous and Partially Homogeneous Environments

A Extended Matrix Cfar Detector With A Pre-Processing Procedure

Modified Jensen-Bregman LogDet Divergence for Target Detection With Region Covariance Descriptor

Geodesic Normal Coordinate-Based Manifold Filtering for Target Detection

Adaptive Detection in Partially Homogeneous Environment with Limited Samples Based on Geometric Barycenters

3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features

An Improved Matrix Information Geometric Detector with Parameter Selection

Coherent Detection of Weak Moving Targets in Compound-Gaussian Clutter Using Nonlinear Preprocessing System: Performance Measure and Implementation

Geometric Means and Medians with Applications to Target Detection.

Target Detection in Sea Clutter Via Weighted Averaging Filter on the Riemannian Manifold

Adaptive multichannel detectors for distributed target based on gradient test

Information Geometry for Radar Target Detection with Total Jensen–Bregman Divergence

Small Target Detection Based on Difference Accumulation and Gaussian Curvature under Complex Conditions

Regularized Polarimetric Target Detection in the Presence of Inhomogeneous And/Or Non-Stationary Clutter