Abstract:With the rapid development of computer graphics and 3D computer vision, the field of reconstructing 3D geometric shapes from sketches has quickly seen an influx of innovative methodologies. However, the majority of current methods primarily rely on the associative memory of matching images to models during their training stages, failing to fully grasp the real structure of the three-dimensional objects presented in the images. These approaches often lead to significant discrepancies between the reconstructed objects and the expected models. For instance, some methods often lose important geometric details when dealing with objects that have complex topological structures, resulting in a significant reduction in the visual consistency between the reconstructed models and the original images. Moreover, existing technologies also demonstrate clear limitations in theoretically capturing the subtle symmetry and local features of three-dimensional objects. To address this issue, we propose a two-phase framework in this work to more accurately reflect the objects in the input images in the reconstructed models. Initially, we employ an encoder–decoder structure to generate an implicit signed distance field (SDF) representing the 3D shapes. Subsequently, we carry out a comprehensive optimization of the decoder from the first phase. This process includes two main steps: firstly, utilizing differentiable rendering techniques to render the mesh model derived from the distance field, ensuring its consistency with the input images; secondly, combining the symmetry of the 3D shapes with innovative regularization loss to further refine the decoder, aiming to reduce the discrepancies between the images and the 3D shapes. Compared to similar research, our method not only demonstrates superior performance in reconstructing 3D shapes from sketches but also offers new perspectives and solutions for the optimization of 3D shapes. This work signifies an important advancement in understanding and reconstructing the transition process from sketches to precise 3D models. Code: https://github.com/Hyuq1/Sketch2Reality.git

High-Fidelity 3D Model Generation with Relightable Appearance from Single Freehand Sketches and Text Guidance

Rapid 3D Model Generation with Intuitive 3D Input

Deep3DSketch+\+: High-Fidelity 3D Modeling from Single Free-hand Sketches

Deep3DSketch+: Rapid 3D Modeling from Single Free-hand Sketches

Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches

Deep3DSketch-im: rapid high-fidelity AI 3D model generation by single freehand sketches

Model-driven sketch reconstruction with structure-oriented retrieval

Sketch-based 3D Modeling by Aligning Outlines of an Image

New Fashion: Personalized 3D Design with a Single Sketch Input

Deep3DSketch+: Obtaining Customized 3D Model by Single Free-Hand Sketch through Deep Learning

Deep3DSketch: 3D modeling from Free-hand Sketches with View- and Structural-Aware Adversarial Training

3D VR Sketch Guided 3D Shape Prototyping and Exploration

Magic3DSketch: Create Colorful 3D Models From Sketch-Based 3D Modeling Guided by Text and Language-Image Pre-Training

SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling

SketchDream: Sketch-based Text-To-3D Generation and Editing

Towards 3D VR-Sketch to 3D Shape Retrieval

FMASketch: Freehand Mid-Air Sketching in AR

Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation

From sketch to reality: precision-friendly 3D generation technology

Reconstructing 3D Model from Single-View Sketch with Deep Neural Network.