Generating Images with 3D Annotations Using Diffusion Models
Wufei Ma,Qihao Liu,Jiahao Wang,Angtian Wang,Xiaoding Yuan,Yi Zhang,Zihao Xiao,Guofeng Zhang,Beijia Lu,Ruxiao Duan,Yongrui Qi,Adam Kortylewski,Yaoyao Liu,Alan Yuille
2024-01-01
Abstract:Diffusion models have emerged as a powerful generative method, capable ofproducing stunning photo-realistic images from natural language descriptions.However, these models lack explicit control over the 3D structure in thegenerated images. Consequently, this hinders our ability to obtain detailed 3Dannotations for the generated images or to craft instances with specific posesand distances. In this paper, we propose 3D Diffusion Style Transfer (3D-DST),which incorporates 3D geometry control into diffusion models. Our methodexploits ControlNet, which extends diffusion models by using visual prompts inaddition to text prompts. We generate images of the 3D objects taken from 3Dshape repositories (e.g., ShapeNet and Objaverse), render them from a varietyof poses and viewing directions, compute the edge maps of the rendered images,and use these edge maps as visual prompts to generate realistic images. Withexplicit 3D geometry control, we can easily change the 3D structures of theobjects in the generated images and obtain ground-truth 3D annotationsautomatically. This allows us to improve a wide range of vision tasks, e.g.,classification and 3D pose estimation, in both in-distribution (ID) andout-of-distribution (OOD) settings. We demonstrate the effectiveness of ourmethod through extensive experiments on ImageNet-100/200, ImageNet-R,PASCAL3D+, ObjectNet3D, and OOD-CV. The results show that our methodsignificantly outperforms existing methods, e.g., 3.8 percentage points onImageNet-100 using DeiT-B.