Precise-Physics Driven Text-to-3D Generation

Qingshan Xu,Jiao Liu,Melvin Wong,Caishun Chen,Yew-Soon Ong

2024-03-19

Abstract:Text-to-3D generation has shown great promise in generating novel 3D content based on given text prompts. However, existing generative methods mostly focus on geometric or visual plausibility while ignoring precise physics perception for the generated 3D shapes. This greatly hinders the practicality of generated 3D shapes in real-world applications. In this work, we propose Phy3DGen, a precise-physics-driven text-to-3D generation method. By analyzing the solid mechanics of generated 3D shapes, we reveal that the 3D shapes generated by existing text-to-3D generation methods are impractical for real-world applications as the generated 3D shapes do not conform to the laws of physics. To this end, we leverage 3D diffusion models to provide 3D shape priors and design a data-driven differentiable physics layer to optimize 3D shape priors with solid mechanics. This allows us to optimize geometry efficiently and learn precise physics information about 3D shapes at the same time. Experimental results demonstrate that our method can consider both geometric plausibility and precise physics perception, further bridging 3D virtual modeling and precise physical worlds.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper proposes a precise physics-driven text-to-3D generation method called Phy3DGen to address the issue of existing 3D shape generation methods ignoring accurate physical perception. While existing text-to-3D generation techniques can generate novel 3D content based on textual prompts, they mostly focus on geometric or visual plausibility and neglect the physical laws that 3D shapes should adhere to, limiting their practicality in real-world applications.

Phy3DGen analyzes the solid mechanics of 3D shapes and finds that the 3D shapes generated by existing methods do not comply with the laws of physics, making them potentially fragile and unsuitable for practical scenarios. To tackle this, the paper proposes leveraging a 3D diffusion model to provide 3D shape priors and designs a data-driven differentiable physics layer to optimize these priors while considering solid mechanics. This way, precise physical information can be learned while optimizing the geometric shape.

Experimental results demonstrate that this method can balance geometric plausibility and accurate physical perception, further connecting 3D virtual modeling with the precise physical world. By combining the 3D diffusion model and the differentiable physics layer, Phy3DGen can optimize the geometric shape during training and learn physical information, ensuring that the generated 3D shapes not only satisfy visual effects but also meet engineering requirements.