Three cheers for the pain pump?

J. Dragoo

DOI: https://doi.org/10.1007/s00167-015-3520-0

2015-02-04

Abstract:

What problem does this paper attempt to address?

Model-driven sketch reconstruction with structure-oriented retrieval

Lei Li,Zhe Huang,Changqing Zou,Chiew-Lan Tai,Rynson W. H. Lau,Hao Zhang,Ping Tan,Hongbo Fu

DOI: https://doi.org/10.1145/3005358.3005372

2016-01-01

Abstract:We propose an interactive system that aims at lifting a 2D sketch into a 3D sketch with the help of existing models in shape collections. The key idea is to exploit part structure for shape retrieval and sketch reconstruction. We adopt sketch-based shape retrieval and develop a novel matching algorithm which considers structure in addition to traditional shape features. From a list of retrieved models, users select one to serve as a 3D proxy, providing abstract 3D information. Then our reconstruction method transforms the sketch into 3D geometry by back-projection, followed by an optimization procedure based on the Laplacian mesh deformation framework. Preliminary evaluations show that our retrieval algorithm is more effective than a state-of-the-art method and users can create interesting 3D forms of sketches without precise drawing skills.
Sketch-based 3D Modeling by Aligning Outlines of an Image

Chunxiao Li,Hyowon Lee,Dongliang Zhang,Hao Jiang

DOI: https://doi.org/10.1016/j.jcde.2016.04.003

2016-01-01

Journal of Computational Design and Engineering

Abstract:In this paper we present an efficient technique for sketch-based 3D modeling using automatically extracted image features. Creating a 3D model often requires a drawing of irregular shapes composed of curved lines as a starting point but it is difficult to hand-draw such lines without introducing awkward bumps and edges along the lines. We propose an automatic alignment of a user's hand-drawn sketch lines to the contour lines of an image, facilitating a considerable level of ease with which the user can carelessly continue sketching while the system intelligently snaps the sketch lines to a background image contour, no longer requiring the strenuous effort and stress of trying to make a perfect line during the modeling task. This interactive technique seamlessly combines the efficiency and perception of the human user with the accuracy of computational power, applied to the domain of 3D modeling where the utmost precision of on-screen drawing has been one of the hurdles of the task hitherto considered a job requiring a highly skilled and careful manipulation by the user. We provide several examples to demonstrate the accuracy and efficiency of the method with which complex shapes were achieved easily and quickly in the interactive outline drawing task. (C) 2016 Society of CAD/CAM Engineers. Publishing Services by Elsevier.
A Comparison of 3D Shape Retrieval Methods Based on a Large-Scale Benchmark Supporting Multimodal Queries

Bo Li,Yijuan Lu,Chunyuan Li,Afzal Godil,Tobias Schreck,Masaki Aono,Martin Burtscher,Qiang Chen,Nihad Karim Chowdhury,Bin Fang,Hongbo Fu,Takahiko Furuya,Haisheng Li,Jianzhuang Liu,Henry Johan,Ryuichi Kosaka,Hitoshi Koyanagi,Ryutarou Ohbuchi,Atsushi Tatsuma,Yajuan Wan,Chaoli Zhang,Changqing Zou

DOI: https://doi.org/10.1016/j.cviu.2014.10.006

IF: 4.886

2015-01-01

Computer Vision and Image Understanding

Abstract:Large-scale 3D shape retrieval has become an important research direction in content-based 3D shape retrieval. To promote this research area, two Shape Retrieval Contest (SHREC) tracks on large scale comprehensive and sketch-based 3D model retrieval have been organized by us in 2014. Both tracks were based on a unified large-scale benchmark that supports multimodal queries (3D models and sketches). This benchmark contains 13680 sketches and 8987 3D models, divided into 171 distinct classes. It was compiled to be a superset of existing benchmarks and presents a new challenge to retrieval methods as it comprises generic models as well as domain-specific model types. Twelve and six distinct 3D shape retrieval methods have competed with each other in these two contests, respectively. To measure and compare the performance of the participating and other promising Query-by-Model or Query-by-Sketch 3D shape retrieval methods and to solicit state-of-the-art approaches, we perform a more comprehensive comparison of twenty-six (eighteen originally participating algorithms and eight additional state-of-the-art or new) retrieval methods by evaluating them on the common benchmark. The benchmark, results, and evaluation tools are publicly available at our websites (http://www.itl.nist.gov/iad/vug/sharp/contest/2014/Generic3D/, 2014, http://www.itl.nist.gov/iad/vug/sharp/contest/2014/SBR/, 2014).
GA-Sketching: Shape Modeling from Multi-View Sketching with Geometry-Aligned Deep Implicit Functions

Jie Zhou,Zhongjin Luo,Qian Yu,Xiaoguang Han,Hongbo Fu

DOI: https://doi.org/10.48550/arXiv.2309.05946

2023-09-12

Abstract:Sketch-based shape modeling aims to bridge the gap between 2D drawing and 3D modeling by providing an intuitive and accessible approach to create 3D shapes from 2D sketches. However, existing methods still suffer from limitations in reconstruction quality and multi-view interaction friendliness, hindering their practical application. This paper proposes a faithful and user-friendly iterative solution to tackle these limitations by learning geometry-aligned deep implicit functions from one or multiple sketches. Our method lifts 2D sketches to volume-based feature tensors, which align strongly with the output 3D shape, enabling accurate reconstruction and faithful editing. Such a geometry-aligned feature encoding technique is well-suited to iterative modeling since features from different viewpoints can be easily memorized or aggregated. Based on these advantages, we design a unified interactive system for sketch-based shape modeling. It enables users to generate the desired geometry iteratively by drawing sketches from any number of viewpoints. In addition, it allows users to edit the generated surface by making a few local modifications. We demonstrate the effectiveness and practicality of our method with extensive experiments and user studies, where we found that our method outperformed existing methods in terms of accuracy, efficiency, and user satisfaction. The source code of this project is available at <a class="link-external link-https" href="https://github.com/LordLiang/GA-Sketching" rel="external noopener nofollow">this https URL</a>.

Graphics
3D VR Sketch Guided 3D Shape Prototyping and Exploration

Ling Luo,Pinaki Nath Chowdhury,Tao Xiang,Yi-Zhe Song,Yulia Gryaditskaya

2024-01-10

Abstract:3D shape modeling is labor-intensive, time-consuming, and requires years of expertise. To facilitate 3D shape modeling, we propose a 3D shape generation network that takes a 3D VR sketch as a condition. We assume that sketches are created by novices without art training and aim to reconstruct geometrically realistic 3D shapes of a given category. To handle potential sketch ambiguity, our method creates multiple 3D shapes that align with the original sketch's structure. We carefully design our method, training the model step-by-step and leveraging multi-modal 3D shape representation to support training with limited training data. To guarantee the realism of generated 3D shapes we leverage the normalizing flow that models the distribution of the latent space of 3D shapes. To encourage the fidelity of the generated 3D shapes to an input sketch, we propose a dedicated loss that we deploy at different stages of the training process. The code is available at <a class="link-external link-https" href="https://github.com/Rowl1ng/3Dsketch2shape" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
From sketch to reality: precision-friendly 3D generation technology

Yuanqi Hu,Jianqi Zhang,Ling Bai,Jing Li,Bing Li,Ying Zang,Wenjun Hu

DOI: https://doi.org/10.1007/s00371-024-03425-z

IF: 2.835

2024-05-18

The Visual Computer

Abstract:With the rapid development of computer graphics and 3D computer vision, the field of reconstructing 3D geometric shapes from sketches has quickly seen an influx of innovative methodologies. However, the majority of current methods primarily rely on the associative memory of matching images to models during their training stages, failing to fully grasp the real structure of the three-dimensional objects presented in the images. These approaches often lead to significant discrepancies between the reconstructed objects and the expected models. For instance, some methods often lose important geometric details when dealing with objects that have complex topological structures, resulting in a significant reduction in the visual consistency between the reconstructed models and the original images. Moreover, existing technologies also demonstrate clear limitations in theoretically capturing the subtle symmetry and local features of three-dimensional objects. To address this issue, we propose a two-phase framework in this work to more accurately reflect the objects in the input images in the reconstructed models. Initially, we employ an encoder–decoder structure to generate an implicit signed distance field (SDF) representing the 3D shapes. Subsequently, we carry out a comprehensive optimization of the decoder from the first phase. This process includes two main steps: firstly, utilizing differentiable rendering techniques to render the mesh model derived from the distance field, ensuring its consistency with the input images; secondly, combining the symmetry of the 3D shapes with innovative regularization loss to further refine the decoder, aiming to reduce the discrepancies between the images and the 3D shapes. Compared to similar research, our method not only demonstrates superior performance in reconstructing 3D shapes from sketches but also offers new perspectives and solutions for the optimization of 3D shapes. This work signifies an important advancement in understanding and reconstructing the transition process from sketches to precise 3D models. Code: https://github.com/Hyuq1/Sketch2Reality.git

computer science, software engineering
DualShape: Sketch-Based 3D Shape Design With Part Generation and Retrieval

Xusheng Du,Tianyu Zhang,Haoran Xie

DOI: https://doi.org/10.1109/access.2024.3361659

IF: 3.9

2024-02-10

IEEE Access

Abstract:Creating a 3D shape design from free-hand sketches is a challenging task due to the sparse and ambiguous information from sketches. In this work, we propose DualShape, a sketch-based 3D shape design interface with part generation and retrieval. DualShape first decouples the model generation into two parts: a composite part retrieval module using a sketch-based feature matching method and a sketch-based part generation module using a deep learning approach with implicit function representation. We then propose an assembly module for the obtained part models to accomplish the task of 3D shape generation from input sketches. In addition, we provide an optimization module that allows users to optimize and manually adjust the assembled model to achieve satisfying models. For the reconstruction interface, shadow guidance was utilized to assist users with the retrieved 3D models matching the input strokes as background references in real time. To verify the effectiveness of the DualShape system, we conducted the comparison experiments and the user study. The results show that DualShape is more user-friendly and is able to generate 3D models with richer details. We believe this work would provide a novel paradigm for hybrid 3D model generation in computer graphics.

computer science, information systems,telecommunications,engineering, electrical & electronic
Deep3DSketch+: Rapid 3D Modeling from Single Free-hand Sketches

Tianrun Chen,Chenglong Fu,Ying Zang,Lanyun Zhu,Jia Zhang,Papa Mao,Lingyun Sun

2023-09-23

Abstract:The rapid development of AR/VR brings tremendous demands for 3D content. While the widely-used Computer-Aided Design (CAD) method requires a time-consuming and labor-intensive modeling process, sketch-based 3D modeling offers a potential solution as a natural form of computer-human interaction. However, the sparsity and ambiguity of sketches make it challenging to generate high-fidelity content reflecting creators' ideas. Precise drawing from multiple views or strategic step-by-step drawings is often required to tackle the challenge but is not friendly to novice users. In this work, we introduce a novel end-to-end approach, Deep3DSketch+, which performs 3D modeling using only a single free-hand sketch without inputting multiple sketches or view information. Specifically, we introduce a lightweight generation network for efficient inference in real-time and a structural-aware adversarial training approach with a Stroke Enhancement Module (SEM) to capture the structural information to facilitate learning of the realistic and fine-detailed shape structures for high-fidelity performance. Extensive experiments demonstrated the effectiveness of our approach with the state-of-the-art (SOTA) performance on both synthetic and real datasets.

Computer Vision and Pattern Recognition
Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation

Aditya Sanghi,Pradeep Kumar Jayaraman,Arianna Rampini,Joseph Lambourne,Hooman Shayani,Evan Atherton,Saeid Asgari Taghanaki

2023-07-08

Abstract:Significant progress has recently been made in creative applications of large pre-trained models for downstream tasks in 3D vision, such as text-to-shape generation. This motivates our investigation of how these pre-trained models can be used effectively to generate 3D shapes from sketches, which has largely remained an open challenge due to the limited sketch-shape paired datasets and the varying level of abstraction in the sketches. We discover that conditioning a 3D generative model on the features (obtained from a frozen large pre-trained vision model) of synthetic renderings during training enables us to effectively generate 3D shapes from sketches at inference time. This suggests that the large pre-trained vision model features carry semantic signals that are resilient to domain shifts, i.e., allowing us to use only RGB renderings, but generalizing to sketches at inference time. We conduct a comprehensive set of experiments investigating different design factors and demonstrate the effectiveness of our straightforward approach for generation of multiple 3D shapes per each input sketch regardless of their level of abstraction without requiring any paired datasets during training.

Computer Vision and Pattern Recognition
Deep3DSketch: 3D modeling from Free-hand Sketches with View- and Structural-Aware Adversarial Training

Tianrun Chen,Chenglong Fu,Lanyun Zhu,Papa Mao,Jia Zhang,Ying Zang,Lingyun Sun

2023-12-08

Abstract:This work aims to investigate the problem of 3D modeling using single free-hand sketches, which is one of the most natural ways we humans express ideas. Although sketch-based 3D modeling can drastically make the 3D modeling process more accessible, the sparsity and ambiguity of sketches bring significant challenges for creating high-fidelity 3D models that reflect the creators' ideas. In this work, we propose a view- and structural-aware deep learning approach, \textit{Deep3DSketch}, which tackles the ambiguity and fully uses sparse information of sketches, emphasizing the structural information. Specifically, we introduced random pose sampling on both 3D shapes and 2D silhouettes, and an adversarial training scheme with an effective progressive discriminator to facilitate learning of the shape structures. Extensive experiments demonstrated the effectiveness of our approach, which outperforms existing methods -- with state-of-the-art (SOTA) performance on both synthetic and real datasets.

Multimedia
Rapid 3D Model Generation with Intuitive 3D Input

Tianrun Chen,Chaotao Ding,Shangzhan Zhang,Chunan Yu,Ying Zang,Zejian Li,Sida Peng,Lingyun Sun

DOI: https://doi.org/10.1109/cvpr52733.2024.01193

2024-01-01

Computer Vision and Pattern Recognition

Abstract:With the emergence of AR/VR, 3D models are in tremendous demand. However, conventional 3D modeling with Computer-Aided Design software requires much expertise and is difficult for novice users. We find that AR/VR devices, in addition to serving as effective display mediums, can offer a promising potential as an intuitive 3D model creation tool, especially with the assistance of AI generative models. Here, we propose Deep3DVRSketch, the first 3D model generation network that inputs 3D VR sketches from novice users and generates highly consistent 3D models in multiple categories within seconds, irrespective of the users' drawing abilities. We also contribute KO3D+, the largest 3D sketch-shape dataset. Our method pre-trains a conditional diffusion model on quality 3D data, then fine-tunes an encoder to map 3D sketches onto the generator's manifold using an adaptive curriculum strategy for limited ground truths. In our experiment, our approach achieves state-of-the-art performance in both model quality and fidelity with real-world input from novice users, and users can even draw and obtain very detailed geometric structures. In our user study, users were able to complete the 3D modeling tasks over 10 times faster using our approach compared to conventional CAD software tools. We believe that our Deep3DVRSketch and KO3D+ dataset can offer a promising solution for future 3D modeling in metaverse era. Check the project page at http://research.kokoni3d.com/Deep3DVRSketch.
Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

Guofei Sun,Yongkang Wong,Mohan S. Kankanhalli,Xiangdong Li,Weidong Geng

DOI: https://doi.org/10.1145/3491224

IF: 4.094

2022-01-01

ACM Transactions on Multimedia Computing Communications and Applications

Abstract:Reconstructing three-dimensional (3D) objects from images has attracted increasing attention due to its wide applications in computer vision and robotic tasks. Despite the promising progress of recent deep learning-based approaches, which directly reconstruct the full 3D shape without considering the conceptual knowledge of the object categories, existing models have limited usage and usually create unrealistic shapes. 3D objects have multiple forms of representation, such as 3D volume, conceptual knowledge, and so on. In this work, we show that the conceptual knowledge for a category of objects, which represents objects as prototype volumes and is structured by graph, can enhance the 3D reconstruction pipeline. We propose a novel multimodal framework that explicitly combines graph-based conceptual knowledge with deep neural networks for 3D shape reconstruction from a single RGB image. Our approach represents conceptual knowledge of a specific category as a structure-based knowledge graph. Specifically, conceptual knowledge acts as visual priors and spatial relationships to assist the 3D reconstruction framework to create realistic 3D shapes with enhanced details. Our 3D reconstruction framework takes an image as input. It first predicts the conceptual knowledge of the object in the image, then generates a 3D object based on the input image and the predicted conceptual knowledge. The generated 3D object satisfies the following requirements: (1) it is consistent with the predicted graph in concept, and (2) consistent with the input image in geometry. Extensive experiments on public datasets (i.e., ShapeNet, Pix3D, and Pascal3D+) with 13 object categories show that (1) our method outperforms the state-of-the-art methods, (2) our prototype volume-based conceptual knowledge representation is more effective, and (3) our pipeline-agnostic approach can enhance the reconstruction quality of various 3D shape reconstruction pipelines.
Deep3DSketch+\+: High-Fidelity 3D Modeling from Single Free-hand Sketches

Ying Zang,Chaotao Ding,Tianrun Chen,Papa Mao,Wenjun Hu

2023-10-27

Abstract:The rise of AR/VR has led to an increased demand for 3D content. However, the traditional method of creating 3D content using Computer-Aided Design (CAD) is a labor-intensive and skill-demanding process, making it difficult to use for novice users. Sketch-based 3D modeling provides a promising solution by leveraging the intuitive nature of human-computer interaction. However, generating high-quality content that accurately reflects the creator's ideas can be challenging due to the sparsity and ambiguity of sketches. Furthermore, novice users often find it challenging to create accurate drawings from multiple perspectives or follow step-by-step instructions in existing methods. To address this, we introduce a groundbreaking end-to-end approach in our work, enabling 3D modeling from a single free-hand sketch, Deep3DSketch+$\backslash$+. The issue of sparsity and ambiguity using single sketch is resolved in our approach by leveraging the symmetry prior and structural-aware shape discriminator. We conducted comprehensive experiments on diverse datasets, including both synthetic and real data, to validate the efficacy of our approach and demonstrate its state-of-the-art (SOTA) performance. Users are also more satisfied with results generated by our approach according to our user study. We believe our approach has the potential to revolutionize the process of 3D modeling by offering an intuitive and easy-to-use solution for novice users.

Human-Computer Interaction
Deep3DSketch+: Obtaining Customized 3D Model by Single Free-Hand Sketch through Deep Learning

Ying Zang,Chenglong Fu,Tianrun Chen,Yuanqi Hu,Qingshan Liu,Wenjun Hu

2023-10-28

Abstract:As 3D models become critical in today's manufacturing and product design, conventional 3D modeling approaches based on Computer-Aided Design (CAD) are labor-intensive, time-consuming, and have high demands on the creators. This work aims to introduce an alternative approach to 3D modeling by utilizing free-hand sketches to obtain desired 3D models. We introduce Deep3DSketch+, which is a deep-learning algorithm that takes the input of a single free-hand sketch and produces a complete and high-fidelity model that matches the sketch input. The neural network has view- and structural-awareness enabled by a Shape Discriminator (SD) and a Stroke Enhancement Module (SEM), which overcomes the limitations of sparsity and ambiguity of the sketches. The network design also brings high robustness to partial sketch input in industrial applications.Our approach has undergone extensive experiments, demonstrating its state-of-the-art (SOTA) performance on both synthetic and real-world datasets. These results validate the effectiveness and superiority of our method compared to existing techniques. We have demonstrated the conversion of free-hand sketches into physical 3D objects using additive manufacturing. We believe that our approach has the potential to accelerate product design and democratize customized manufacturing.

Multimedia
Sketch-based 3D Shape Modeling from Sparse Point Clouds

Xusheng Du,Yi He,Xi Yang,Chia-Ming Chang,Haoran Xie

DOI: https://doi.org/10.48550/arXiv.2201.11287

2022-01-27

Abstract:3D modeling based on point clouds is an efficient way to reconstruct and create detailed 3D content. However, the geometric procedure may lose accuracy due to high redundancy and the absence of an explicit structure. In this work, we propose a human-in-the-loop sketch-based point cloud reconstruction framework to leverage users cognitive abilities in geometry extraction. We present an interactive drawing interface for 3D model creation from point cloud data with the help of user sketches. We adopt an optimization method in which the user can continuously edit the contours extracted from the obtained 3D model and retrieve the model iteratively. Finally, we verify the proposed user interface for modeling from sparse point clouds. see video here <a class="link-external link-https" href="https://www.youtube.com/watch?v=0H19NyXDRJE" rel="external noopener nofollow">this https URL</a> .

Graphics,Human-Computer Interaction
Towards 3D VR-Sketch to 3D Shape Retrieval

Ling Luo,Yulia Gryaditskaya,Yongxin Yang,Tao Xiang,Yi-Zhe Song

DOI: https://doi.org/10.48550/arXiv.2209.10020

2022-09-20

Computer Vision and Pattern Recognition

Abstract:Growing free online 3D shapes collections dictated research on 3D retrieval. Active debate has however been had on (i) what the best input modality is to trigger retrieval, and (ii) the ultimate usage scenario for such retrieval. In this paper, we offer a different perspective towards answering these questions -- we study the use of 3D sketches as an input modality and advocate a VR-scenario where retrieval is conducted. Thus, the ultimate vision is that users can freely retrieve a 3D model by air-doodling in a VR environment. As a first stab at this new 3D VR-sketch to 3D shape retrieval problem, we make four contributions. First, we code a VR utility to collect 3D VR-sketches and conduct retrieval. Second, we collect the first set of $167$ 3D VR-sketches on two shape categories from ModelNet. Third, we propose a novel approach to generate a synthetic dataset of human-like 3D sketches of different abstract levels to train deep networks. At last, we compare the common multi-view and volumetric approaches: We show that, in contrast to 3D shape to 3D shape retrieval, volumetric point-based approaches exhibit superior performance on 3D sketch to 3D shape retrieval due to the sparse and abstract nature of 3D VR-sketches. We believe these contributions will collectively serve as enablers for future attempts at this problem. The VR interface, code and datasets are available at https://tinyurl.com/3DSketch3DV.
Discriminative Sketch-Based 3d Model Retrieval Via Robust Shape Matching

Tianjia Shao,Weiwei Xu,Kangkang Yin,Jingdong Wang,Kun Zhou,Baining Guo

DOI: https://doi.org/10.1111/j.1467-8659.2011.02050.x

IF: 2.5

2011-01-01

Computer Graphics Forum

Abstract:We propose a sketch-based 3D shape retrieval system that is substantially more discriminative and robust than existing systems, especially for complex models. The power of our system comes from a combination of a contour-based 2D shape representation and a robust sampling-based shape matching scheme. They are defined over discriminative local features and applicable for partial sketches; robust to noise and distortions in hand drawings; and consistent when strokes are added progressively. Our robust shape matching, however, requires dense sampling and registration and incurs a high computational cost. We thus devise critical acceleration methods to achieve interactive performance: precomputing kNN graphs that record transformations between neighboring contour images and enable fast online shape alignment; pruning sampling and shape registration strategically and hierarchically; and parallelizing shape matching on multi-core platforms or GPUs. We demonstrate the effectiveness of our system through various experiments, comparisons, and user studies.
Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation

Wangguandong Zheng,Haifeng Xia,Rui Chen,Ming Shao,Siyu Xia,Zhengming Ding

2024-04-07

Abstract:Recently, image-to-3D approaches have achieved significant results with a natural image as input. However, it is not always possible to access these enriched color input samples in practical applications, where only sketches are available. Existing sketch-to-3D researches suffer from limitations in broad applications due to the challenges of lacking color information and multi-view content. To overcome them, this paper proposes a novel generation paradigm Sketch3D to generate realistic 3D assets with shape aligned with the input sketch and color matching the textual description. Concretely, Sketch3D first instantiates the given sketch in the reference image through the shape-preserving generation process. Second, the reference image is leveraged to deduce a coarse 3D Gaussian prior, and multi-view style-consistent guidance images are generated based on the renderings of the 3D Gaussians. Finally, three strategies are designed to optimize 3D Gaussians, i.e., structural optimization via a distribution transfer mechanism, color optimization with a straightforward MSE loss and sketch similarity optimization with a CLIP-based geometric similarity loss. Extensive visual comparisons and quantitative analysis illustrate the advantage of our Sketch3D in generating realistic 3D assets while preserving consistency with the input.

Computer Vision and Pattern Recognition
Deep Learning Assisted Optimization for 3D Reconstruction from Single 2D Line Drawings

Jia Zheng,Yifan Zhu,Kehan Wang,Qiang Zou,Zihan Zhou

2022-01-01

Abstract: In this paper, we revisit the long-standing problem of automatic reconstruction of 3D objects from single line drawings. Previous optimization-based methods can generate compact and accurate 3D models, but their success rates depend heavily on the ability to (i) identifying a sufficient set of true geometric constraints, and (ii) choosing a good initial value for the numerical optimization. In view of these challenges, we propose to train deep neural networks to detect pairwise relationships among geometric entities (i.e., edges) in the 3D object, and to predict initial depth value of the vertices. Our experiments on a large dataset of CAD models show that, by leveraging deep learning in a geometric constraint solving pipeline, the success rate of optimization-based 3D reconstruction can be significantly improved.
Deep Optimized Priors for 3D Shape Modeling and Reconstruction

Mingyue Yang,Yuxin Wen,Weikai Chen,Yongwei Chen,Kui Jia

DOI: https://doi.org/10.48550/arXiv.2012.07241

2020-12-14

Abstract:Many learning-based approaches have difficulty scaling to unseen data, as the generality of its learned prior is limited to the scale and variations of the training samples. This holds particularly true with 3D learning tasks, given the sparsity of 3D datasets available. We introduce a new learning framework for 3D modeling and reconstruction that greatly improves the generalization ability of a deep generator. Our approach strives to connect the good ends of both learning-based and optimization-based methods. In particular, unlike the common practice that fixes the pre-trained priors at test time, we propose to further optimize the learned prior and latent code according to the input physical measurements after the training. We show that the proposed strategy effectively breaks the barriers constrained by the pre-trained priors and could lead to high-quality adaptation to unseen data. We realize our framework using the implicit surface representation and validate the efficacy of our approach in a variety of challenging tasks that take highly sparse or collapsed observations as input. Experimental results show that our approach compares favorably with the state-of-the-art methods in terms of both generality and accuracy.

Computer Vision and Pattern Recognition

Three cheers for the pain pump?

Model-driven sketch reconstruction with structure-oriented retrieval

Sketch-based 3D Modeling by Aligning Outlines of an Image

A Comparison of 3D Shape Retrieval Methods Based on a Large-Scale Benchmark Supporting Multimodal Queries

GA-Sketching: Shape Modeling from Multi-View Sketching with Geometry-Aligned Deep Implicit Functions

3D VR Sketch Guided 3D Shape Prototyping and Exploration

From sketch to reality: precision-friendly 3D generation technology

DualShape: Sketch-Based 3D Shape Design With Part Generation and Retrieval

Deep3DSketch+: Rapid 3D Modeling from Single Free-hand Sketches

Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation

Deep3DSketch: 3D modeling from Free-hand Sketches with View- and Structural-Aware Adversarial Training

Rapid 3D Model Generation with Intuitive 3D Input

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

Deep3DSketch+\+: High-Fidelity 3D Modeling from Single Free-hand Sketches

Deep3DSketch+: Obtaining Customized 3D Model by Single Free-Hand Sketch through Deep Learning

Sketch-based 3D Shape Modeling from Sparse Point Clouds

Towards 3D VR-Sketch to 3D Shape Retrieval

Discriminative Sketch-Based 3d Model Retrieval Via Robust Shape Matching

Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation

Deep Learning Assisted Optimization for 3D Reconstruction from Single 2D Line Drawings

Deep Optimized Priors for 3D Shape Modeling and Reconstruction