Abstract:There has emerged substantial research in addressing single-view 3D reconstruction and the majority of the state-of-the-art implicit methods employ CNNs as the backbone network. On the other hand, transformers have shown remarkable performance in many vision tasks. However, it is still unknown whether transformers are suitable for single-view implicit 3D reconstruction. In this paper, we propose the first end-to-end single-view 3D reconstruction network based on the Pyramid Vision Transformer (PVT), called ED2IF2-Net, which disentangles the reconstruction of an implicit field into the reconstruction of topological structures and the recovery of surface details to achieve high-fidelity shape reconstruction. ED2IF2-Net uses a Pyramid Vision Transformer encoder to extract multi-scale hierarchical local features and a global vector of the input single image, which are fed into three separate decoders. A coarse shape decoder reconstructs a coarse implicit field based on the global vector, a deformation decoder iteratively refines the coarse implicit field using the pixel-aligned local features to obtain a deformed implicit field through multiple implicit field deformation blocks (IFDBs), and a surface detail decoder predicts an enhanced displacement field using the local features with hybrid attention modules (HAMs). The final output is a fusion of the deformed implicit field and the enhanced displacement field, with four loss terms applied to reconstruct the coarse implicit field, structure details through a novel deformation loss, overall shape after fusion, and surface details via a Laplacian loss. The quantitative results obtained from the ShapeNet dataset validate the exceptional performance of ED2IF2-Net. Notably, ED2IF2-Net-L stands out as the top-performing variant, exhibiting the highest mean IoU, CD, EMD, ECD-3D, and ECD-2D scores, reaching impressive values of 61.1, 7.26, 2.51, 6.08, and 1.84, respectively. The extensive experimental evaluations consistently demonstrate the state-of-the-art capabilities of ED2IF2-Net in terms of reconstructing topological structures and recovering surface details, all while maintaining competitive inference time.

Visual and Spatial Context Fusion for Implicit Human Reconstruction

ED2IF2-Net: Learning Disentangled Deformed Implicit Fields and Enhanced Displacement Fields from Single Images Using Pyramid Vision Transformer

Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction

Disambiguating Monocular Reconstruction of 3D Clothed Human with Spatial-Temporal Transformer

3D Human Reconstruction from A Single Depth Image

UNIF: United Neural Implicit Functions for Clothed Human Reconstruction and Animation

Implicit 3D Human Reconstruction Guided by Parametric Models and Normal Maps

SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video

SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction

Multi‐Level Implicit Function for Detailed Human Reconstruction by Relaxing SMPL Constraints

Pixel2ISDF: Implicit Signed Distance Fields based Human Body Model from Multi-view and Multi-pose Images

Learning Pose Controllable Human Reconstruction with Dynamic Implicit Fields from a Single Image

SeSDF: Self-evolved Signed Distance Field for Implicit 3D Clothed Human Reconstruction

RICH: Robust Implicit Clothed Humans Reconstruction from Multi-scale Spatial Cues.

Self-Calibrated Neural Implicit 3D Reconstruction

AdaptiveFusion: Adaptive Multi-Modal Multi-View Fusion for 3D Human Body Reconstruction

Deep Implicit Templates for 3D Shape Representation

Implicit Neural Representations With Structured Latent Codes for Human Body Modeling

Local Deep Implicit Functions for 3D Shape

Detailed 3D Human Body Reconstruction from Multi-view Images Combining Voxel Super-Resolution and Learned Implicit Representation