Abstract:Recently, the emergence of diffusion models has opened up new opportunities for single-view reconstruction. However, all the existing methods represent the target object as a closed mesh devoid of any structural information, thus neglecting the part-based structure, which is crucial for many downstream applications, of the reconstructed shape. Moreover, the generated meshes usually suffer from large noises, unsmooth surfaces, and blurry textures, making it challenging to obtain satisfactory part segments using 3D segmentation techniques. In this paper, we present Part123, a novel framework for part-aware 3D reconstruction from a single-view image. We first use diffusion models to generate multiview-consistent images from a given image, and then leverage Segment Anything Model (SAM), which demonstrates powerful generalization ability on arbitrary objects, to generate multiview segmentation masks. To effectively incorporate 2D part-based information into 3D reconstruction and handle inconsistency, we introduce contrastive learning into a neural rendering framework to learn a part-aware feature space based on the multiview segmentation masks. A clustering-based algorithm is also developed to automatically derive 3D part segmentation results from the reconstructed models. Experiments show that our method can generate 3D models with high-quality segmented parts on various objects. Compared to existing unstructured reconstruction methods, the part-aware 3D models from our method benefit some important applications, including feature-preserving reconstruction, primitive fitting, and 3D shape editing.

PGNet: A Part-based Generative Network for 3D Object Reconstruction

3D-Mask-GAN:Unsupervised Single-View 3D Object Reconstruction

3D Reconstruction and Semantic Segmentation Method Combining PointNet and 3D-Lmnet from Single Image

Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction

CGNet: A Cascaded Generative Network for dense point cloud reconstruction from a single image

3D Point Cloud Generation Reconstruction from Single Image Based on Image Retrieval

G3DR: Generative 3D Reconstruction in ImageNet

Part123: Part-aware 3D Reconstruction from a Single-view Image

MVPNet: Multi-View Point Regression Networks for 3D Object Reconstruction from A Single Image

Part-Wise AtlasNet for 3D Point Cloud Reconstruction from a Single Image

A Coarse-to-Fine Transformer-Based Network for 3D Reconstruction from Non-Overlapping Multi-View Images

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Semi-supervised Three-dimensional Reconstruction Framework with GAN.

PolyGNN: Polyhedron-based Graph Neural Network for 3D Building Reconstruction from Point Clouds

3D-RVP: A method for 3D object reconstruction from a single depth view using voxel and point

Learning Progressive Point Embeddings for 3D Point Cloud Generation

StarNet: Style-Aware 3D Point Cloud Generation

PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation

Single-Image 3-D Reconstruction: Rethinking Point Cloud Deformation

Progressive Learning of 3D Reconstruction Network from 2D GAN Data

DV-Net: Dual-view Network for 3D Reconstruction by Fusing Multiple Sets of Gated Control Point Clouds