FlexiDreamer: Single Image-to-3D Generation with FlexiCubes

Ruowen Zhao,Zhengyi Wang,Yikai Wang,Zihan Zhou,Jun Zhu

2024-05-27

Abstract:3D content generation has wide applications in various fields. One of its dominant paradigms is by sparse-view reconstruction using multi-view images generated by diffusion models. However, since directly reconstructing triangle meshes from multi-view images is challenging, most methodologies opt to an implicit representation (such as NeRF) during the sparse-view reconstruction and acquire the target mesh by a post-processing extraction. However, the implicit representation takes extensive time to train and the post-extraction also leads to undesirable visual artifacts. In this paper, we propose FlexiDreamer, a novel framework that directly reconstructs high-quality meshes from multi-view generated images. We utilize an advanced gradient-based mesh optimization, namely FlexiCubes, for multi-view mesh reconstruction, which enables us to generate 3D meshes in an end-to-end manner. To address the reconstruction artifacts owing to the inconsistencies from generated images, we design a hybrid positional encoding scheme to improve the reconstruction geometry and an orientation-aware texture mapping to mitigate surface ghosting. To further enhance the results, we respectively incorporate eikonal and smooth regularizations to reduce geometric holes and surface noise. Our approach can generate high-fidelity 3D meshes in the single image-to-3D downstream task with approximately 1 minute, significantly outperforming previous methods.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper introduces FlexiDreamer, a new framework for generating high-precision 3D meshes from a single image. Current methods typically rely on multi-view reconstruction and neural volume representations (such as NeRF), but these methods suffer from long training times and post-processing artifacts. FlexiDreamer achieves an end-to-end process by introducing advanced gradient-based mesh optimization method called FlexiCubes to directly reconstruct high-quality 3D meshes from multiple views. To address the issue of inconsistent reconstructions caused by multiple view images, the paper proposes a hybrid positional encoding to improve geometric reconstruction and designs direction-aware texture mapping to mitigate surface ghosting. In addition, eikonal and smooth regularization are applied to reduce geometric holes and surface noise. FlexiDreamer can generate high-fidelity 3D texture meshes in approximately 1 minute, significantly outperforming existing methods.