Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering

Qijun Gan,Wentong Li,Jinwei Ren,Jianke Zhu
2024-07-09
Abstract:Reconstructing high-fidelity hand models with intricate textures plays a crucial role in enhancing human-object interaction and advancing real-world applications. Despite the state-of-the-art methods excelling in texture generation and image rendering, they often face challenges in accurately capturing geometric details. Learning-based approaches usually offer better robustness and faster inference, which tend to produce smoother results and require substantial amounts of training data. To address these issues, we present a novel fine-grained multi-view hand mesh reconstruction method that leverages inverse rendering to restore hand poses and intricate details. Firstly, our approach predicts a parametric hand mesh model through Graph Convolutional Networks (GCN) based method from multi-view images. We further introduce a novel Hand Albedo and Mesh (HAM) optimization module to refine both the hand mesh and textures, which is capable of preserving the mesh topology. In addition, we suggest an effective mesh-based neural rendering scheme to simultaneously generate photo-realistic image and optimize mesh geometry by fusing the pre-trained rendering network with vertex features. We conduct the comprehensive experiments on InterHand2.6M, DeepHandMesh and dataset collected by ourself, whose promising results show that our proposed approach outperforms the state-of-the-art methods on both reconstruction accuracy and rendering quality. Code and dataset are publicly available at <a class="link-external link-https" href="https://github.com/agnJason/FMHR" rel="external noopener nofollow">this https URL</a>.
Computer Vision and Pattern Recognition,Artificial Intelligence
What problem does this paper attempt to address?
This paper attempts to solve the problems of high - precision geometric details and texture generation in hand reconstruction. Although existing methods perform well in texture generation and image rendering, they still face challenges in accurately capturing geometric details. Learning - based methods, while providing better robustness and faster inference speed, usually produce smoother results and require a large amount of training data. For this reason, the paper proposes a new fine - grained multi - view hand mesh reconstruction method, which uses inverse rendering technology to recover hand poses and complex details. Specifically, the main contributions of the paper include: 1. Propose a coarse - to - fine method for accurately recovering a fine - grained hand mesh model from multi - view images, using inverse rendering technology. 2. Introduce a new Hand Albedo and Mesh (HAM) optimization module to refine the over - smoothed results of the parameterized hand model. 3. Design an effective mesh - based neural rendering scheme that simultaneously generates realistic images and optimizes mesh geometry by fusing a pre - trained rendering network with vertex features. The paper verifies the effectiveness of the proposed method through experiments on InterHand2.6M, DeepHandMesh, and a self - collected dataset. The experimental results show that this method is superior to existing methods in both reconstruction accuracy and rendering quality.