Abstract:Hand avatars play a pivotal role in a wide array of digital interfaces, enhancing user immersion and facilitating natural interaction within virtual environments. While previous studies have focused on photo-realistic hand rendering, little attention has been paid to reconstruct the hand geometry with fine details, which is essential to rendering quality. In the realms of extended reality and gaming, on-the-fly rendering becomes imperative. To this end, we introduce an expressive hand avatar, named XHand, that is designed to comprehensively generate hand shape, appearance, and deformations in real-time. To obtain fine-grained hand meshes, we make use of three feature embedding modules to predict hand deformation displacements, albedo, and linear blending skinning weights, respectively. To achieve photo-realistic hand rendering on fine-grained meshes, our method employs a mesh-based neural renderer by leveraging mesh topological consistency and latent codes from embedding modules. During training, a part-aware Laplace smoothing strategy is proposed by incorporating the distinct levels of regularization to effectively maintain the necessary details and eliminate the undesired artifacts. The experimental evaluations on InterHand2.6M and DeepHandMesh datasets demonstrate the efficacy of XHand, which is able to recover high-fidelity geometry and texture for hand animations across diverse poses in real-time. To reproduce our results, we will make the full implementation publicly available at <a class="link-external link-https" href="https://github.com/agnJason/XHand" rel="external noopener nofollow">this https URL</a>.

MediaPipe Hands: On-device Real-time Hand Tracking

Lightweight real-time hand segmentation leveraging MediaPipe landmark detection

Real-Time Human Pose Detection and Recognition Using MediaPipe

On-device Real-time Custom Hand Gesture Recognition

Optimizing Hand Region Detection in MediaPipe Holistic Full-Body Pose Estimation to Improve Accuracy and Avoid Downstream Errors

Real-time pose and shape reconstruction of two interacting hands with a single depth camera

Estimate Hand Poses Efficiently from Single Depth Images

Efficient Human Pose Estimation: Leveraging Advanced Techniques with MediaPipe

RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video

Ar3dHands: A Dataset and Baseline for Real-Time 3D Hand Pose Estimation from Binocular Distorted Images

Real-Time Hand Gesture Monitoring Model Based on MediaPipe's Registerable System

Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data

EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream

XHand: Real-time Expressive Hand Avatar

Using a single RGB frame for real time 3D hand pose estimation in the wild

FastHand: Fast monocular hand pose estimation on embedded systems

HandDGP: Camera-Space Hand Mesh Prediction with Differentiable Global Positioning

Fast and Robust Hand Tracking Using Detection-Guided Optimization

MotionTrace: IMU-based Field of View Prediction for Smartphone AR Interactions

Testing MediaPipe Holistic for Linguistic Analysis of Nonmanual Markers in Sign Languages

Next-Gen Dynamic Hand Gesture Recognition: MediaPipe, Inception-v3 and LSTM-Based Enhanced Deep Learning Model