Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering

Yibo Zhang,Lihong Wang,Changqing Zou,Tieru Wu,Rui Ma

2024-05-24

Abstract:3D sketches are widely used for visually representing the 3D shape and structure of objects or scenes. However, the creation of 3D sketch often requires users to possess professional artistic skills. Existing research efforts primarily focus on enhancing the ability of interactive sketch generation in 3D virtual systems. In this work, we propose Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch by optimizing 3D parametric curves under various supervisions. Specifically, we perform perspective projection to render the 3D rational Bézier curves into 2D curves, which are subsequently converted to a 2D raster image via our customized differentiable rasterizer. Our framework bridges the domains of 3D sketch and raster image, achieving end-toend optimization of 3D sketch through gradients computed in the 2D image domain. Our Diff3DS can enable a series of novel 3D sketch generation tasks, including textto-3D sketch and image-to-3D sketch, supported by the popular distillation-based supervision, such as Score Distillation Sampling (SDS). Extensive experiments have yielded promising results and demonstrated the potential of our framework.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of generating perspective-consistent 3D sketches. Specifically, existing 3D sketch generation methods mainly focus on interactively generating sketches in 3D virtual systems, which usually require users to have professional artistic skills and 3D operation experience, making them not user-friendly for the general public. Additionally, existing multi-view image-based methods (such as 3Doodle) have some limitations when generating 3D sketches, such as ignoring the depth order of curves and requiring multi-view image supervision.

To overcome these issues, the authors propose a novel differentiable rendering framework called Diff3DS, which can generate perspective-consistent 3D sketches from flexible inputs (such as a single image or text). Specifically, Diff3DS generates 3D sketches by optimizing 3D parametric curves and converts 3D rational Bézier curves into 2D rational Bézier curves through perspective projection, then uses a custom differentiable rasterizer to render them into 2D images. This framework supports end-to-end 3D curve optimization and leverages pre-trained 2D image generation models (such as Score Distillation Sampling) for supervision, thereby enabling the task of generating 3D sketches from text or a single image.

In summary, this paper aims to provide a user-friendly, perspective-consistent 3D sketch generation method to lower the threshold for 3D sketch generation, allowing ordinary users to easily create high-quality 3D sketches.