Abstract:Accurately recovering the dense 3D mesh of both hands from monocular images poses considerable challenges due to occlusions and projection ambiguity. Most of the existing methods extract features from color images to estimate the root-aligned hand meshes, which neglect the crucial depth and scale information in the real world. Given the noisy sensor measurements with limited resolution, depth-based methods predict 3D keypoints rather than a dense mesh. These limitations motivate us to take advantage of these two complementary inputs to acquire dense hand meshes on a real-world scale. In this work, we propose an end-to-end framework for recovering dense meshes for both hands, which employ single-view RGB-D image pairs as input. The primary challenge lies in effectively utilizing two different input modalities to mitigate the blurring effects in RGB images and noises in depth images. Instead of directly treating depth maps as additional channels for RGB images, we encode the depth information into the unordered point cloud to preserve more geometric details. Specifically, our framework employs ResNet50 and PointNet++ to derive features from RGB and point cloud, respectively. Additionally, we introduce a novel pyramid deep fusion network (PDFNet) to aggregate features at different scales, which demonstrates superior efficacy compared to previous fusion strategies. Furthermore, we employ a GCN-based decoder to process the fused features and recover the corresponding 3D pose and dense mesh. Through comprehensive ablation experiments, we have not only demonstrated the effectiveness of our proposed fusion algorithm but also outperformed the state-of-the-art approaches on publicly available datasets. To reproduce the results, we will make our source code and models publicly available at https://github.com/zijinxuxu/PDFNet.

DV-Net: Dual-view Network for 3D Reconstruction by Fusing Multiple Sets of Gated Control Point Clouds

Dual-scale Point Cloud Completion Network Based on High-Frequency Feature Fusion

View-Aware Geometry-Structure Joint Learning for Single-View 3D Shape Reconstruction

Dual-View 3D Reconstruction via Learning Correspondence and Dependency of Point Cloud Regions

3D Reconstruction for Multi-view Objects

Dense Point Cloud Reconstruction by Shape and Pose Features Learning

Attention-Guided Fusion Network of Point Cloud and Multiple Views for 3D Shape Recognition.

Multi-view Fusion with Deep Learning for 3D Shape Classification

Learning To Reconstruct High-Quality 3d Shapes With Cascaded Fully Convolutional Networks

Single-view 3D Reconstruction Algorithm Based on View-aware

DmifNet:3D Shape Reconstruction Based on Dynamic Multi-Branch Information Fusion

High-Quality Textured 3D Shape Reconstruction with Cascaded Fully Convolutional Networks

A Single Stage and Single View 3D Point Cloud Reconstruction Network Based on DetNet

DFNet: Deep Fusion Network for 3D Shape Classification and Retrieval

PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition.

A Coarse-to-Fine Transformer-Based Network for 3D Reconstruction from Non-Overlapping Multi-View Images

PVRNet: Point-View Relation Neural Network for 3D Shape Recognition

Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images

Three-Dimensional Reconstruction of Single Input Image Based on Point Cloud

VIPNet: A Fast and Accurate Single-View Volumetric Reconstruction by Learning Sparse Implicit Point Guidance