Abstract:We propose PICASSO, a novel framework CAD sketch parameterization from hand-drawn or precise sketch images via rendering self-supervision. Given a drawing of a CAD sketch, the proposed framework turns it into parametric primitives that can be imported into CAD software. Compared to existing methods, PICASSO enables the learning of parametric CAD sketches from either precise or hand-drawn sketch images, even in cases where annotations at the parameter level are scarce or unavailable. This is achieved by leveraging the geometric characteristics of sketches as a learning cue to pre-train a CAD parameterization network. Specifically, PICASSO comprises two primary components: (1) a Sketch Parameterization Network (SPN) that predicts a series of parametric primitives from CAD sketch images, and (2) a Sketch Rendering Network (SRN) that renders parametric CAD sketches in a differentiable manner. SRN facilitates the computation of a image-to-image loss, which can be utilized to pre-train SPN, thereby enabling zero- and few-shot learning scenarios for the parameterization of hand-drawn sketches. Extensive evaluation on the widely used SketchGraphs dataset validates the effectiveness of the proposed framework.

What problem does this paper attempt to address?

The paper is primarily dedicated to addressing the problem of automatically extracting parametric information from hand-drawn or precise Computer-Aided Design (CAD) sketch images. Specifically, the research proposes a new framework named PICASSO, which aims to achieve zero-shot and few-shot learning for CAD sketch parameterization through a rendering self-supervised approach. ### Research Background and Objectives Traditional CAD workflows typically require designers to convert hand-drawn conceptual sketches into parametric CAD sketches, a process that is both time-consuming and requires a high level of expertise. Therefore, automating this parameterization process has garnered significant attention from both academia and industry. However, extracting CAD sketch parameters from images (especially hand-drawn images) poses numerous challenges, including the vast solution space, the complexity of sketch designs, and the inaccuracies of hand-drawn images. ### Main Contributions 1. **Image-Level Pretraining**: PICASSO is an innovative framework that can perform image-level pretraining directly from precise or hand-drawn CAD sketch images, thereby achieving CAD sketch parameterization in zero-shot and few-shot learning scenarios for the first time. 2. **Differentiable Renderer**: A neural differentiable renderer named SRN (Sketch Rendering Network) is proposed to render CAD parametric primitives into images. SRN enables the use of image-to-image loss to pretrain the CAD sketch parameterization network SPN. 3. **Non-Autoregressive Parameterization Network**: SPN (Sketch Parametrization Network) is a feedforward network that, unlike autoregressive methods, can predict parametric primitives of CAD sketches in a non-autoregressive manner. This approach has demonstrated state-of-the-art performance in experiments. 4. **Comprehensive Evaluation**: PICASSO has been comprehensively evaluated both qualitatively and quantitatively on the widely used SketchGraphs dataset, particularly demonstrating its effectiveness in zero-shot and few-shot settings. ### Solution Overview - **SRN (Sketch Rendering Network)**: Responsible for rendering parametric CAD primitives into images. SRN allows for the computation of image-level loss between the predicted image and the input image through neural differentiable rendering techniques, thereby supporting the pretraining of SPN. - **SPN (Sketch Parametrization Network)**: Responsible for predicting a set of parametric primitives from CAD sketch images. SPN adopts a feedforward network architecture, avoiding the sequential dependency issues present in autoregressive methods. - **Zero-Shot and Few-Shot Learning**: Through the rendering self-supervised mechanism provided by SRN, SPN can be pretrained with limited or even no parametric-level annotations, enabling it to effectively parameterize hand-drawn or precise CAD sketch images in zero-shot and few-shot learning scenarios.

PICASSO: A Feed-Forward Framework for Parametric Inference of CAD Sketches via Rendering Self-Supervision

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

SmartCanvas: Context-inferred Interpretation of Sketches for Preparatory Design Studies.

SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches

Sketchformer++: A Hierarchical Transformer Architecture for Vector Sketch Representation

DAVINCI: A Single-Stage Architecture for Constrained CAD Sketch Inference

Sketch2CAD: Sequential CAD Modeling by Sketching in Context

SECAD-Net: Self-Supervised CAD Reconstruction by Learning Sketch-Extrude Operations

Parametric Primitive Analysis of CAD Sketches with Vision Transformer

CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention

Sketch Vision: Artificial Intelligence with Sight for Imagination

Deep3DSketch+: Rapid 3D Modeling from Single Free-hand Sketches

Learning to Sketch with Shortcut Cycle Consistency

Sketch-guided Image Inpainting with Partial Discrete Diffusion Process

SketchGraphs: A Large-Scale Dataset for Modeling Relational Geometry in Computer-Aided Design

SketchINR: A First Look into Sketches as Implicit Neural Representations

'CADSketchNet' -- An Annotated Sketch dataset for 3D CAD Model Retrieval with Deep Neural Networks

Sketch-Specific Data Augmentation for Freehand Sketch Recognition

FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context

Deep3DSketch-im: rapid high-fidelity AI 3D model generation by single freehand sketches

Learning Dense Correspondences between Photos and Sketches