Abstract:Reverse engineering in the realm of Computer-Aided Design (CAD) has been a longstanding aspiration, though not yet entirely realized. Its primary aim is to uncover the CAD process behind a physical object given its 3D scan. We propose CAD-SIGNet, an end-to-end trainable and auto-regressive architecture to recover the design history of a CAD model represented as a sequence of sketch-and-extrusion from an input point cloud. Our model learns visual-language representations by layer-wise cross-attention between point cloud and CAD language embedding. In particular, a new Sketch instance Guided Attention (SGA) module is proposed in order to reconstruct the fine-grained details of the sketches. Thanks to its auto-regressive nature, CAD-SIGNet not only reconstructs a unique full design history of the corresponding CAD model given an input point cloud but also provides multiple plausible design choices. This allows for an interactive reverse engineering scenario by providing designers with multiple next-step choices along with the design process. Extensive experiments on publicly available CAD datasets showcase the effectiveness of our approach against existing baseline models in two settings, namely, full design history recovery and conditional auto-completion from point clouds.

What problem does this paper attempt to address?

### Problem the Paper Attempts to Solve This paper attempts to solve the problem of recovering the design history of computer-aided design (CAD) models from point cloud data. Specifically, the paper proposes an end-to-end trainable and autoregressive architecture named CAD-SIGNet, which aims to recover the design history of CAD models from input point cloud data. These design histories are represented in the form of a series of sketches and extrusion operations. ### Background and Motivation In the field of computer-aided design (CAD), reverse engineering has been a long-term goal. Its main objective is to reveal the underlying CAD design process given the 3D scan data of a physical object. Existing methods primarily focus on decomposing the point cloud into parametric primitives to reconstruct the final CAD model. However, CAD modeling is a sequential process where designers need to draw 2D sketches and apply CAD operations (such as extrusion, chamfering, etc.). Recovering these intermediate design steps is crucial for editing and reusing different parts of objects with the same functionality. For example, a chair can be composed of 3 design steps: legs, seat, and backrest. Recovering these steps allows for editing the height of the legs or reusing the backrest in another chair design. ### Main Contributions 1. **End-to-end trainable autoregressive network**: CAD-SIGNet is the first network to propose an autoregressive strategy to infer CAD language from point clouds. 2. **Multimodal Transformer blocks**: Introduces inter-layer cross-attention mechanisms for joint learning between point cloud and CAD language embeddings. 3. **Sketch Instance Guided Attention (SGA) module**: This module guides the inter-layer cross-attention mechanism to focus on relevant regions in the point cloud to predict sketch parameters. 4. **Extensive experimental validation**: Experiments were conducted in two different reverse engineering scenarios, including complete design history recovery and point cloud-based conditional auto-completion. ### Method Overview The core of CAD-SIGNet lies in the joint learning of point cloud and CAD language representations through multimodal Transformer blocks. Each Transformer block includes inter-layer cross-attention mechanisms, enabling the network to extract visual information from the point cloud and combine it with CAD language representations. Specifically, the SGA module improves the accuracy of fine-grained sketch inference by selecting specific subsets (i.e., sketch instances) from the point cloud. ### Experimental Results Experimental results on the DeepCAD dataset show that CAD-SIGNet significantly outperforms existing DeepCAD and MultiCAD methods in the design history recovery task. Specifically, the results are as follows: - **Chamfer distance**: The median Chamfer distance of CAD-SIGNet is 35 times and 28 times lower than that of DeepCAD and MultiCAD, respectively. - **Invalid model ratio**: The proportion of valid models generated by the CAD sequences predicted by CAD-SIGNet is much higher than that of DeepCAD and MultiCAD. - **F1 score**: CAD-SIGNet is more accurate in predicting basic shape types and their occurrences in the design sequence, especially in the F1 score of arc types, which is over 14% higher than DeepCAD. ### Conclusion By introducing an autoregressive strategy and multimodal Transformer blocks, CAD-SIGNet successfully addresses the problem of recovering the design history of CAD models from point cloud data. Experimental results demonstrate that this method has significant advantages in reverse engineering tasks, capable of generating more accurate and valid CAD models.

CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention

SECAD-Net: Self-Supervised CAD Reconstruction by Learning Sketch-Extrude Operations

Point2CAD: Reverse Engineering CAD Models from 3D Point Clouds

CAD Cut-piece Retrieval Method Based on Representation Learning

PS-CAD: Local Geometry Guidance via Prompting and Selection for CAD Reconstruction

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

Computer-Aided Design as Language

'CADSketchNet' -- An Annotated Sketch dataset for 3D CAD Model Retrieval with Deep Neural Networks

CAD-Recode: Reverse Engineering CAD Code from Point Clouds

CAD-GPT: Synthesising CAD Construction Sequence with Spatial Reasoning-Enhanced Multimodal LLMs

Sketch2CAD: Sequential CAD Modeling by Sketching in Context

Automatic Reverse Engineering: Creating computer-aided design (CAD) models from multi-view images

Self-supervised Graph Neural Network for Mechanical CAD Retrieval

Inferring CAD Modeling Sequences Using Zone Graphs

NeurCADRecon: Neural Representation for Reconstructing CAD Surfaces by Enforcing Zero Gaussian Curvature

CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches

CAD-LLM: Large Language Model for CAD Generation

DAVINCI: A Single-Stage Architecture for Constrained CAD Sketch Inference

Extending Point-Based Deep Learning Approaches for Better Semantic Segmentation in CAD

TransCAD: A Hierarchical Transformer for CAD Sequence Inference from Point Clouds