Abstract:There has emerged substantial research in addressing single-view 3D reconstruction and the majority of the state-of-the-art implicit methods employ CNNs as the backbone network. On the other hand, transformers have shown remarkable performance in many vision tasks. However, it is still unknown whether transformers are suitable for single-view implicit 3D reconstruction. In this paper, we propose the first end-to-end single-view 3D reconstruction network based on the Pyramid Vision Transformer (PVT), called ED2IF2-Net, which disentangles the reconstruction of an implicit field into the reconstruction of topological structures and the recovery of surface details to achieve high-fidelity shape reconstruction. ED2IF2-Net uses a Pyramid Vision Transformer encoder to extract multi-scale hierarchical local features and a global vector of the input single image, which are fed into three separate decoders. A coarse shape decoder reconstructs a coarse implicit field based on the global vector, a deformation decoder iteratively refines the coarse implicit field using the pixel-aligned local features to obtain a deformed implicit field through multiple implicit field deformation blocks (IFDBs), and a surface detail decoder predicts an enhanced displacement field using the local features with hybrid attention modules (HAMs). The final output is a fusion of the deformed implicit field and the enhanced displacement field, with four loss terms applied to reconstruct the coarse implicit field, structure details through a novel deformation loss, overall shape after fusion, and surface details via a Laplacian loss. The quantitative results obtained from the ShapeNet dataset validate the exceptional performance of ED2IF2-Net. Notably, ED2IF2-Net-L stands out as the top-performing variant, exhibiting the highest mean IoU, CD, EMD, ECD-3D, and ECD-2D scores, reaching impressive values of 61.1, 7.26, 2.51, 6.08, and 1.84, respectively. The extensive experimental evaluations consistently demonstrate the state-of-the-art capabilities of ED2IF2-Net in terms of reconstructing topological structures and recovering surface details, all while maintaining competitive inference time.

DmifNet:3D Shape Reconstruction Based on Dynamic Multi-Branch Information Fusion

DV-Net: Dual-view Network for 3D Reconstruction by Fusing Multiple Sets of Gated Control Point Clouds

3D Reconstruction and Semantic Segmentation Method Combining PointNet and 3D-Lmnet from Single Image

Reconstructing Recognizable 3D Face Shapes Based on 3D Morphable Models.

DMNet: Delaunay Meshing Network for 3D Shape Representation

Geometry-aware Two-scale PIFu Representation for Human Reconstruction

DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image

3D-Isrnet

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

DTF-Net: Category-Level Pose Estimation and Shape Reconstruction via Deformable Template Field

DPF-Net: Combining Explicit Shape Priors in Deformable Primitive Field for Unsupervised Structural Reconstruction of 3D Objects

D$^2$IM-Net: Learning Detail Disentangled Implicit Fields from Single Images

Shape registration with learned deformations for 3D shape reconstruction from sparse and incomplete point clouds

DAN: Deep-Attention Network for 3D Shape Recognition

Topology-Preserving Shape Reconstruction and Registration via Neural Diffeomorphic Flow

From 2D Images to 3D Model:Weakly Supervised Multi-View Face Reconstruction with Deep Fusion

ED2IF2-Net: Learning Disentangled Deformed Implicit Fields and Enhanced Displacement Fields from Single Images Using Pyramid Vision Transformer

STD-Net: Structure-preserving and Topology-adaptive Deformation Network for Single-View 3D Reconstruction

High-Quality Textured 3D Shape Reconstruction with Cascaded Fully Convolutional Networks

HDR-Net-Fusion: Real-time 3D dynamic scene reconstruction with a hierarchical deep reinforcement network

Image-Guided Human Reconstruction via Multi-Scale Graph Transformation Networks