Abstract:Accurately recovering the dense 3D mesh of both hands from monocular images poses considerable challenges due to occlusions and projection ambiguity. Most of the existing methods extract features from color images to estimate the root-aligned hand meshes, which neglect the crucial depth and scale information in the real world. Given the noisy sensor measurements with limited resolution, depth-based methods predict 3D keypoints rather than a dense mesh. These limitations motivate us to take advantage of these two complementary inputs to acquire dense hand meshes on a real-world scale. In this work, we propose an end-to-end framework for recovering dense meshes for both hands, which employ single-view RGB-D image pairs as input. The primary challenge lies in effectively utilizing two different input modalities to mitigate the blurring effects in RGB images and noises in depth images. Instead of directly treating depth maps as additional channels for RGB images, we encode the depth information into the unordered point cloud to preserve more geometric details. Specifically, our framework employs ResNet50 and PointNet++ to derive features from RGB and point cloud, respectively. Additionally, we introduce a novel pyramid deep fusion network (PDFNet) to aggregate features at different scales, which demonstrates superior efficacy compared to previous fusion strategies. Furthermore, we employ a GCN-based decoder to process the fused features and recover the corresponding 3D pose and dense mesh. Through comprehensive ablation experiments, we have not only demonstrated the effectiveness of our proposed fusion algorithm but also outperformed the state-of-the-art approaches on publicly available datasets. To reproduce the results, we will make our source code and models publicly available at https://github.com/zijinxuxu/PDFNet.

Region-Aware Dynamic Filtering Network for 3D Hand Reconstruction

In-Hand 3D Object Reconstruction from a Monocular RGB Video

Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image

CAMInterHand: Cooperative Attention for Multi-View Interactive Hand Pose and Mesh Reconstruction

Robust 3D Reconstruction with an RGB-D Camera

HandS3C: 3D Hand Mesh Reconstruction with State Space Spatial Channel Attention from RGB images

Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB Video

3D Hand Mesh Recovery from Monocular RGB in Camera Space

NCRF: Neural Contact Radiance Fields for Free-Viewpoint Rendering of Hand-Object Interaction

3D hand reconstruction via aggregating intra and inter graphs guided by prior knowledge for hand-object interaction scenario

HandNeRF: Learning to Reconstruct Hand-Object Interaction Scene from a Single RGB Image

High Fidelity 3D Hand Shape Reconstruction via Scalable Graph Frequency Decomposition

Joint Hand-Object 3D Reconstruction From a Single Image With Cross-Branch Feature Fusion

RealisticHands: A Hybrid Model for 3D Hand Reconstruction

ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction

Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images

HandOS: 3D Hand Reconstruction in One Stage

DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Field