Abstract:Photorealistic 3D vehicle models with high controllability are essential for autonomous driving simulation and data augmentation. While handcrafted CAD models provide flexible controllability, free CAD libraries often lack the high-quality materials necessary for photorealistic rendering. Conversely, reconstructed 3D models offer high-fidelity rendering but lack controllability. In this work, we introduce UrbanCAD, a framework that pushes the frontier of the photorealism-controllability trade-off by generating highly controllable and photorealistic 3D vehicle digital twins from a single urban image and a collection of free 3D CAD models and handcrafted materials. These digital twins enable realistic 360-degree rendering, vehicle insertion, material transfer, relighting, and component manipulation such as opening doors and rolling down windows, supporting the construction of long-tail scenarios. To achieve this, we propose a novel pipeline that operates in a retrieval-optimization manner, adapting to observational data while preserving flexible controllability and fine-grained handcrafted details. Furthermore, given multi-view background perspective and fisheye images, we approximate environment lighting using fisheye images and reconstruct the background with 3DGS, enabling the photorealistic insertion of optimized CAD models into rendered novel view backgrounds. Experimental results demonstrate that UrbanCAD outperforms baselines based on reconstruction and retrieval in terms of photorealism. Additionally, we show that various perception models maintain their accuracy when evaluated on UrbanCAD with in-distribution configurations but degrade when applied to realistic out-of-distribution data generated by our method. This suggests that UrbanCAD is a significant advancement in creating photorealistic, safety-critical driving scenarios for downstream applications.

Reconstructing vehicles from orthographic drawings using deep neural networks

Mending Neural Implicit Modeling for 3D Vehicle Reconstruction in the Wild

Reconstructing Vechicles from a Single Image: Shape Priors for Road Scene Understanding

Probabilistic Vehicle Reconstruction Using a Multi-Task CNN

Automatic Reverse Engineering: Creating computer-aided design (CAD) models from multi-view images

Neural Rendering based Urban Scene Reconstruction for Autonomous Driving

NeuRodin: A Two-stage Framework for High-Fidelity Neural Surface Reconstruction

Crowd-Sourced NeRF: Collecting Data from Production Vehicles for 3D Street View Reconstruction

Analysis of AI-Based Single-View 3D Reconstruction Methods for an Industrial Application

Neural-PBIR Reconstruction of Shape, Material, and Illumination

Vehicle Reconstruction and Texture Estimation Using Deep Implicit Semantic Template Mapping

Active Implicit Reconstruction Using One-Shot View Planning

Dynamic Multi-View Scene Reconstruction Using Neural Implicit Surface

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Res-NeuS: Deep Residuals and Neural Implicit Surface Learning for Multi-View Reconstruction

MV-DeepSDF: Implicit Modeling with Multi-Sweep Point Clouds for 3D Vehicle Reconstruction in Autonomous Driving

Reconstructing Objects In-the-wild for Realistic Sensor Simulation

UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulation

Deep Single-View 3D Object Reconstruction with Visual Hull Embedding

DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstruction

AutoRecon: Automated 3D Object Discovery and Reconstruction.