Abstract:Reconstructing high-fidelity hand models with intricate textures plays a crucial role in enhancing human-object interaction and advancing real-world applications. Despite the state-of-the-art methods excelling in texture generation and image rendering, they often face challenges in accurately capturing geometric details. Learning-based approaches usually offer better robustness and faster inference, which tend to produce smoother results and require substantial amounts of training data. To address these issues, we present a novel fine-grained multi-view hand mesh reconstruction method that leverages inverse rendering to restore hand poses and intricate details. Firstly, our approach predicts a parametric hand mesh model through Graph Convolutional Networks (GCN) based method from multi-view images. We further introduce a novel Hand Albedo and Mesh (HAM) optimization module to refine both the hand mesh and textures, which is capable of preserving the mesh topology. In addition, we suggest an effective mesh-based neural rendering scheme to simultaneously generate photo-realistic image and optimize mesh geometry by fusing the pre-trained rendering network with vertex features. We conduct the comprehensive experiments on InterHand2.6M, DeepHandMesh and dataset collected by ourself, whose promising results show that our proposed approach outperforms the state-of-the-art methods on both reconstruction accuracy and rendering quality. Code and dataset are publicly available at <a class="link-external link-https" href="https://github.com/agnJason/FMHR" rel="external noopener nofollow">this https URL</a>.

Differentiable Rendering Approach to Mesh Optimization for Digital Human Reconstruction.

Multiview Textured Mesh Recovery by Differentiable Rendering

A General Differentiable Mesh Renderer for Image-Based 3D Reasoning

High-fidelity 3D Face Reconstruction with Multi-Scale Details

Efficient Multi-View Inverse Rendering Using a Hybrid Differentiable Rendering Method

Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering

Simplified and Tessellated Mesh for Realtime High Quality Rendering.

Analytical Derivatives for Differentiable Renderer: 3D Pose Estimation by Silhouette Consistency

Monocular Human Pose and Shape Reconstruction Using Part Differentiable Rendering

Single-View 3D Reconstruction via Differentiable Rendering and Inverse Procedural Modeling

HDhuman: High-quality Human Novel-view Rendering from Sparse Views

Deformable Model-Driven Neural Rendering for High-Fidelity 3D Reconstruction of Human Heads Under Low-View Settings

DRWR: A Differentiable Renderer Without Rendering for Unsupervised 3D Structure Learning from Silhouette Images

HDhuman: High-quality Human Performance Capture with Sparse Views

Towards Efficient and Photorealistic 3D Human Reconstruction: A Brief Survey

Seamless images stitching for 3D human models

Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

PSHuman: Photorealistic Single-view Human Reconstruction using Cross-Scale Diffusion

Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering

HISR: Hybrid Implicit Surface Representation for Photorealistic 3D Human Reconstruction