Abstract:Sketch-based 3D shape retrieval (SBSR) can be approached by learning domain-invariant descriptors or ranking metrics from sketches and 2D view images of 3D shapes rendered through numerous viewpoints. However, determining the most appropriate viewpoints that convey discriminative geometric features to benefit the task of SBSR became an essential yet not fully explored area. Existing works extract 3D features from multi-view images observed through pre-defined viewpoints to match 2D sketches. Those methods, however, fail to dynamically select viewpoints by considering the SBSR task. In this work, we introduce a fully differentiable viewpoint learning paradigm driven by the downstream SBSR task, which supports the task-aware and sketch-dependent dynamic viewpoint determination process. We naturally integrate this task-specific and sketch-dependent viewpoint learning process into a meta-learning framework to develop a novel Dynamic Viewer (DV) module for SBSR. DV module comprises a Meta View Learner (MVL) block and a View Generator (VG) block. Specifically, as the first part of the DV module, the MVL block learns to initiate the necessary network parameters of the VG block. Then, the VG block that serves as the second part learns the best viewpoints to render 2D images. To learn the optimal viewpoints for SBSR, we further introduce a view mining loss that aims to maximize the similarity of feature-level information among rendered 2D views and the query sketch. Further, we adopt a variational autoencoder (VAE) to retrieve 3D shapes by setting the newly rendered images and query sketch as inputs. As evidenced by the comprehensive experimental results conducted on popular SBSR datasets, the proposed framework has been demonstrated to outperform recent methods in both category-level sketch-based and fine-grained SBSR.

Learning best views of 3D shapes from sketch contour

Training Deep Convolutional Neural Networks to Acquire the Best View of a 3D Shape.

Viewpoint-Aware Representation for Sketch-Based 3D Model Retrieval

Sketch-based 3D Modeling by Aligning Outlines of an Image

Model-driven sketch reconstruction with structure-oriented retrieval

Shape Representation and Recognition in High Dimensional Feature Space

Sketch-Based 3D Model Retrieval via Multi-feature Fusion

Retrieval-Specific View Learning for Sketch-to-Shape Retrieval

Towards 3D VR-Sketch to 3D Shape Retrieval

GA-Sketching: Shape Modeling from Multi-View Sketching with Geometry-Aligned Deep Implicit Functions

Sketch based modeling and editing via shape space exploration

3D Sketching for 3D Object Retrieval

Sketch-based 3D Model Retrieval Utilizing Adaptive View Clustering and Semantic Information

Structure-aware Viewpoint Selection for Volume Visualization

A Unified Feature Representation and Learning Framework for 3D Shape

Scribble-Based 3D Shape Segmentation via Weakly-Supervised Learning

Deep3DSketch: 3D modeling from Free-hand Sketches with View- and Structural-Aware Adversarial Training

Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches.

Structure-Aware 3D VR Sketch to 3D Shape Retrieval

Learning View Selection for 3D Scenes.

Deep Sketch-Shape Hashing With Segmented 3D Stochastic Viewing